Oct 30 21:13:02.697756 ip-10-0-129-83 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:13:03.141200 ip-10-0-129-83 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:13:03.141200 ip-10-0-129-83 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:13:03.141200 ip-10-0-129-83 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:13:03.141200 ip-10-0-129-83 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:13:03.148101 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148021 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:13:03.148101 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148099 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:13:03.148101 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148103 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:13:03.148101 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148106 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:13:03.148101 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148109 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:13:03.148245 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148112 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:13:03.148245 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148115 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:13:03.148245 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148117 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:13:03.148245 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148120 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:13:03.148245 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148122 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:13:03.148245 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148125 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:13:03.148245 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148130 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:13:03.148245 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148134 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:13:03.148245 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148137 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:13:03.148245 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148140 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:13:03.148245 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148143 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:13:03.148245 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148146 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:13:03.148245 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148149 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:13:03.148245 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148158 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:13:03.148245 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148161 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:13:03.148245 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148163 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:13:03.148245 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148166 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:13:03.148245 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148169 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:13:03.148245 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148171 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:13:03.148245 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148174 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:13:03.148758 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148177 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:13:03.148758 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148182 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:13:03.148758 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148185 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:13:03.148758 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148189 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:13:03.148758 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148192 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:13:03.148758 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148195 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:13:03.148758 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148197 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:13:03.148758 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148200 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:13:03.148758 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148202 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:13:03.148758 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148205 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:13:03.148758 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148208 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:13:03.148758 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148210 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:13:03.148758 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148213 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:13:03.148758 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148215 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:13:03.148758 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148218 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:13:03.148758 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148221 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:13:03.148758 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148224 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:13:03.148758 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148226 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:13:03.148758 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148229 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:13:03.149199 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148231 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:13:03.149199 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148234 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:13:03.149199 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148236 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:13:03.149199 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148239 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:13:03.149199 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148242 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:13:03.149199 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148244 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:13:03.149199 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148247 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:13:03.149199 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148250 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:13:03.149199 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148253 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:13:03.149199 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148256 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:13:03.149199 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148259 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:13:03.149199 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148262 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:13:03.149199 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148264 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:13:03.149199 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148267 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:13:03.149199 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148269 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:13:03.149199 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148272 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:13:03.149199 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148275 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:13:03.149199 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148278 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:13:03.149199 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148280 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:13:03.149199 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148283 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:13:03.149714 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148285 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:13:03.149714 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148288 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:13:03.149714 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148290 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:13:03.149714 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148293 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:13:03.149714 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148295 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:13:03.149714 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148298 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:13:03.149714 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148301 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:13:03.149714 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148304 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:13:03.149714 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148307 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:13:03.149714 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148310 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:13:03.149714 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148312 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:13:03.149714 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148315 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:13:03.149714 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148317 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:13:03.149714 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148320 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:13:03.149714 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148322 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:13:03.149714 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148326 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:13:03.149714 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148329 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:13:03.149714 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148331 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:13:03.149714 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148334 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:13:03.150158 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148337 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:13:03.150158 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148340 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:13:03.150158 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148343 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:13:03.150158 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148346 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:13:03.150158 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148349 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:13:03.150158 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148352 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:13:03.150158 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148354 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:13:03.150158 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148358 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:13:03.150158 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148360 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:13:03.150158 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148363 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:13:03.150158 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148366 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:13:03.150158 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148368 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:13:03.150158 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148371 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:13:03.150158 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148374 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:13:03.150158 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148377 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:13:03.150158 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148379 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:13:03.150158 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148382 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:13:03.150158 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148384 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:13:03.150158 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148387 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:13:03.150158 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148390 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:13:03.150712 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148393 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:13:03.150712 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148395 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:13:03.150712 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.148398 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:13:03.150712 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149843 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:13:03.150712 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149852 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:13:03.150712 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149855 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:13:03.150712 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149858 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:13:03.150712 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149861 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:13:03.150712 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149864 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:13:03.150712 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149867 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:13:03.150712 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149869 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:13:03.150712 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149872 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:13:03.150712 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149874 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:13:03.150712 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149877 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:13:03.150712 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149880 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:13:03.150712 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149884 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:13:03.150712 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149887 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:13:03.150712 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149889 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:13:03.150712 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149892 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:13:03.150712 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149895 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:13:03.151205 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149897 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:13:03.151205 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149900 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:13:03.151205 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149903 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:13:03.151205 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149905 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:13:03.151205 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149908 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:13:03.151205 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149911 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:13:03.151205 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149914 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:13:03.151205 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149916 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:13:03.151205 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149919 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:13:03.151205 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149921 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:13:03.151205 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149924 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:13:03.151205 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149927 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:13:03.151205 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149931 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:13:03.151205 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149935 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:13:03.151205 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149938 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:13:03.151205 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149941 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:13:03.151205 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149944 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:13:03.151205 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149946 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:13:03.151205 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149949 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:13:03.151205 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149951 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:13:03.151698 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149954 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:13:03.151698 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149956 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:13:03.151698 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149959 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:13:03.151698 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149961 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:13:03.151698 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149964 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:13:03.151698 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149967 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:13:03.151698 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149970 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:13:03.151698 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149973 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:13:03.151698 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149975 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:13:03.151698 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149978 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:13:03.151698 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149980 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:13:03.151698 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149983 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:13:03.151698 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149986 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:13:03.151698 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149988 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:13:03.151698 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149991 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:13:03.151698 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149993 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:13:03.151698 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149996 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:13:03.151698 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.149998 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:13:03.151698 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150001 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:13:03.152205 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150003 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:13:03.152205 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150006 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:13:03.152205 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150008 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:13:03.152205 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150011 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:13:03.152205 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150013 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:13:03.152205 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150016 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:13:03.152205 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150019 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:13:03.152205 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150021 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:13:03.152205 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150024 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:13:03.152205 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150026 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:13:03.152205 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150029 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:13:03.152205 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150032 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:13:03.152205 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150034 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:13:03.152205 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150037 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:13:03.152205 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150040 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:13:03.152205 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150042 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:13:03.152205 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150045 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:13:03.152205 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150048 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:13:03.152205 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150050 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:13:03.152674 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150053 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:13:03.152674 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150055 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:13:03.152674 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150058 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:13:03.152674 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150061 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:13:03.152674 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150063 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:13:03.152674 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150066 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:13:03.152674 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150068 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:13:03.152674 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150071 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:13:03.152674 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150073 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:13:03.152674 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150076 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:13:03.152674 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150078 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:13:03.152674 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150081 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:13:03.152674 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150083 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:13:03.152674 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150086 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:13:03.152674 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150088 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:13:03.152674 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150091 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:13:03.152674 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150094 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:13:03.152674 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150097 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:13:03.152674 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150102 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:13:03.153125 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150105 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:13:03.153125 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150108 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:13:03.153125 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150111 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:13:03.153125 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150114 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:13:03.153125 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150117 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:13:03.153125 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150120 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:13:03.153125 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150122 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:13:03.153125 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150125 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:13:03.153125 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150127 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:13:03.153125 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150130 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:13:03.153125 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150133 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:13:03.153125 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.150137 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:13:03.153125 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151584 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:13:03.153125 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151592 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:13:03.153125 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151595 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:13:03.153125 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151600 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:13:03.153125 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151602 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:13:03.153125 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151606 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:13:03.153125 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151608 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:13:03.153125 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151611 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:13:03.153616 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151614 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:13:03.153616 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151616 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:13:03.153616 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151619 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:13:03.153616 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151622 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:13:03.153616 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151625 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:13:03.153616 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151627 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:13:03.153616 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151630 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:13:03.153616 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151632 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:13:03.153616 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151636 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:13:03.153616 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151639 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:13:03.153616 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151641 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:13:03.153616 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151644 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:13:03.153616 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151646 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:13:03.153616 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151649 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:13:03.153616 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151651 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:13:03.153616 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151654 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:13:03.153616 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151656 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:13:03.153616 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151659 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:13:03.153616 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151662 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:13:03.153616 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151664 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:13:03.154068 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151667 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:13:03.154068 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151669 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:13:03.154068 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151671 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:13:03.154068 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151674 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:13:03.154068 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151676 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:13:03.154068 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151679 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:13:03.154068 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151681 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:13:03.154068 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151684 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:13:03.154068 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151687 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:13:03.154068 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151690 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:13:03.154068 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151692 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:13:03.154068 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151695 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:13:03.154068 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151697 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:13:03.154068 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151700 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:13:03.154068 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151702 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:13:03.154068 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151705 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:13:03.154068 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151707 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:13:03.154068 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151710 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:13:03.154068 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151713 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:13:03.154068 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151715 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:13:03.154541 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151718 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:13:03.154541 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151720 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:13:03.154541 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151723 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:13:03.154541 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151725 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:13:03.154541 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151729 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:13:03.154541 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151732 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:13:03.154541 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151734 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:13:03.154541 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151737 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:13:03.154541 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151739 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:13:03.154541 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151742 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:13:03.154541 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151745 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:13:03.154541 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151747 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:13:03.154541 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151749 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:13:03.154541 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151752 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:13:03.154541 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151754 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:13:03.154541 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151757 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:13:03.154541 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151759 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:13:03.154541 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151761 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:13:03.154541 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151764 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:13:03.154541 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151766 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:13:03.155007 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151769 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:13:03.155007 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151772 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:13:03.155007 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151775 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:13:03.155007 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151777 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:13:03.155007 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151780 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:13:03.155007 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151782 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:13:03.155007 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151784 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:13:03.155007 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151787 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:13:03.155007 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151789 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:13:03.155007 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151792 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:13:03.155007 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151795 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:13:03.155007 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151799 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:13:03.155007 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151802 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:13:03.155007 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151804 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:13:03.155007 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151807 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:13:03.155007 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151810 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:13:03.155007 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151812 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:13:03.155007 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151815 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:13:03.155007 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151817 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:13:03.155476 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151820 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:13:03.155476 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151822 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:13:03.155476 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151825 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:13:03.155476 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151827 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:13:03.155476 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151832 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:13:03.155476 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151836 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:13:03.155476 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151839 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:13:03.155476 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151842 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:13:03.155476 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151845 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:13:03.155476 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151847 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:13:03.155476 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151850 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:13:03.155476 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151852 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:13:03.155476 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151854 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:13:03.155476 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151857 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:13:03.155476 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151859 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:13:03.155476 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151862 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:13:03.155476 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151865 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:13:03.155476 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151867 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:13:03.155476 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.151870 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:13:03.155921 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152795 2577 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:13:03.155921 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152805 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:13:03.155921 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152812 2577 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:13:03.155921 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152817 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:13:03.155921 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152821 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:13:03.155921 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152824 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:13:03.155921 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152829 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:13:03.155921 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152833 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:13:03.155921 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152836 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:13:03.155921 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152839 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:13:03.155921 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152843 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:13:03.155921 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152846 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:13:03.155921 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152849 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:13:03.155921 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152852 2577 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:13:03.155921 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152855 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:13:03.155921 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152858 2577 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:13:03.155921 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152861 2577 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:13:03.155921 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152864 2577 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:13:03.155921 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152870 2577 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:13:03.155921 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152873 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:13:03.155921 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152876 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:13:03.155921 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152879 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:13:03.155921 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152883 2577 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:13:03.155921 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152887 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:13:03.156524 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152890 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:13:03.156524 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152893 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:13:03.156524 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152896 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:13:03.156524 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152899 2577 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:13:03.156524 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152902 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:13:03.156524 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152906 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:13:03.156524 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152911 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:13:03.156524 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152914 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:13:03.156524 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152919 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:13:03.156524 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152922 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:13:03.156524 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152925 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:13:03.156524 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152928 2577 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:13:03.156524 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152931 2577 flags.go:64] FLAG: --enable-server="true" Oct 30 21:13:03.156524 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152934 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:13:03.156524 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152940 2577 flags.go:64] FLAG: --event-burst="100" Oct 30 21:13:03.156524 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152943 2577 flags.go:64] FLAG: --event-qps="50" Oct 30 21:13:03.156524 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152946 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:13:03.156524 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152950 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:13:03.156524 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152953 2577 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:13:03.156524 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152958 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:13:03.156524 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152961 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:13:03.156524 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152964 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:13:03.156524 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152968 2577 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:13:03.156524 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152971 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:13:03.156524 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152974 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:13:03.157150 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152977 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:13:03.157150 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152980 2577 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:13:03.157150 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152983 2577 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:13:03.157150 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152986 2577 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:13:03.157150 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152989 2577 flags.go:64] FLAG: --feature-gates="" Oct 30 21:13:03.157150 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152993 2577 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:13:03.157150 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.152996 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:13:03.157150 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153000 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:13:03.157150 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153003 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:13:03.157150 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153006 2577 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:13:03.157150 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153009 2577 flags.go:64] FLAG: --help="false" Oct 30 21:13:03.157150 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153012 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:03.157150 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153016 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:13:03.157150 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153021 2577 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:13:03.157150 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153024 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:13:03.157150 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153028 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:13:03.157150 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153031 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:13:03.157150 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153034 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:13:03.157150 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153037 2577 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:13:03.157150 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153040 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:13:03.157150 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153043 2577 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:13:03.157150 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153048 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:13:03.157150 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153051 2577 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:13:03.157720 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153054 2577 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:13:03.157720 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153057 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:13:03.157720 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153060 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:13:03.157720 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153063 2577 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:13:03.157720 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153065 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:13:03.157720 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153069 2577 flags.go:64] FLAG: --lock-file="" Oct 30 21:13:03.157720 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153071 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:13:03.157720 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153074 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:13:03.157720 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153077 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:13:03.157720 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153083 2577 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:13:03.157720 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153086 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:13:03.157720 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153089 2577 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:13:03.157720 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153093 2577 flags.go:64] FLAG: --logging-format="text" Oct 30 21:13:03.157720 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153096 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:13:03.157720 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153099 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:13:03.157720 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153102 2577 flags.go:64] FLAG: --manifest-url="" Oct 30 21:13:03.157720 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153105 2577 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:13:03.157720 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153110 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:13:03.157720 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153113 2577 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:13:03.157720 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153117 2577 flags.go:64] FLAG: --max-pods="110" Oct 30 21:13:03.157720 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153120 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:13:03.157720 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153123 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:13:03.157720 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153127 2577 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:13:03.157720 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153131 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:13:03.157720 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153134 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:13:03.158311 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153137 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:13:03.158311 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153140 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:13:03.158311 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153148 2577 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:13:03.158311 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153151 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:13:03.158311 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153154 2577 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:13:03.158311 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153159 2577 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:13:03.158311 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153162 2577 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:13:03.158311 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153165 2577 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:13:03.158311 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153168 2577 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:13:03.158311 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153171 2577 flags.go:64] FLAG: --port="10250" Oct 30 21:13:03.158311 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153175 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:13:03.158311 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153178 2577 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0e1ea79f4fa19c526" Oct 30 21:13:03.158311 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153181 2577 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:13:03.158311 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153184 2577 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:13:03.158311 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153187 2577 flags.go:64] FLAG: --register-node="true" Oct 30 21:13:03.158311 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153190 2577 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:13:03.158311 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153194 2577 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:13:03.158311 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153197 2577 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:13:03.158311 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153200 2577 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:13:03.158311 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153204 2577 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:13:03.158311 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153208 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:13:03.158311 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153211 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:13:03.158311 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153214 2577 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:13:03.158311 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153217 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:13:03.158311 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153220 2577 flags.go:64] FLAG: --runonce="false" Oct 30 21:13:03.158311 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153223 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:13:03.158926 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153227 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:13:03.158926 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153230 2577 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:13:03.158926 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153233 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:13:03.158926 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153236 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:13:03.158926 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153240 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:13:03.158926 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153243 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:13:03.158926 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153246 2577 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:13:03.158926 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153250 2577 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:13:03.158926 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153253 2577 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:13:03.158926 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153255 2577 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:13:03.158926 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153258 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:13:03.158926 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153263 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:13:03.158926 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153266 2577 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:13:03.158926 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153269 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:13:03.158926 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153275 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:13:03.158926 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153278 2577 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:13:03.158926 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153281 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:13:03.158926 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153289 2577 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:13:03.158926 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153292 2577 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:13:03.158926 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153295 2577 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:13:03.158926 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153298 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:13:03.158926 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153301 2577 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:13:03.158926 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153305 2577 flags.go:64] FLAG: --v="2" Oct 30 21:13:03.158926 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153309 2577 flags.go:64] FLAG: --version="false" Oct 30 21:13:03.158926 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153313 2577 flags.go:64] FLAG: --vmodule="" Oct 30 21:13:03.159625 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153317 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:13:03.159625 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153321 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:13:03.159625 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153470 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:13:03.159625 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153474 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:13:03.159625 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153478 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:13:03.159625 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153480 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:13:03.159625 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153483 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:13:03.159625 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153486 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:13:03.159625 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153489 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:13:03.159625 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153491 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:13:03.159625 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153494 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:13:03.159625 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153496 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:13:03.159625 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153499 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:13:03.159625 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153502 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:13:03.159625 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153505 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:13:03.159625 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153508 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:13:03.159625 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153510 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:13:03.159625 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153513 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:13:03.159625 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153517 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:13:03.159625 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153520 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:13:03.160104 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153522 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:13:03.160104 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153525 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:13:03.160104 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153528 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:13:03.160104 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153531 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:13:03.160104 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153534 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:13:03.160104 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153536 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:13:03.160104 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153539 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:13:03.160104 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153542 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:13:03.160104 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153544 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:13:03.160104 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153547 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:13:03.160104 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153550 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:13:03.160104 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153552 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:13:03.160104 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153555 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:13:03.160104 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153558 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:13:03.160104 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153560 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:13:03.160104 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153563 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:13:03.160104 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153565 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:13:03.160104 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153568 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:13:03.160104 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153571 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:13:03.160104 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153573 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:13:03.160700 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153576 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:13:03.160700 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153578 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:13:03.160700 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153581 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:13:03.160700 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153584 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:13:03.160700 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153586 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:13:03.160700 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153589 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:13:03.160700 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153592 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:13:03.160700 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153596 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:13:03.160700 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153599 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:13:03.160700 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153603 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:13:03.160700 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153607 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:13:03.160700 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153610 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:13:03.160700 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153613 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:13:03.160700 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153616 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:13:03.160700 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153619 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:13:03.160700 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153622 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:13:03.160700 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153625 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:13:03.160700 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153628 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:13:03.160700 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153631 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:13:03.161497 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153634 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:13:03.161497 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153637 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:13:03.161497 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153639 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:13:03.161497 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153642 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:13:03.161497 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153644 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:13:03.161497 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153647 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:13:03.161497 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153649 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:13:03.161497 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153652 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:13:03.161497 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153655 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:13:03.161497 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153657 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:13:03.161497 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153660 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:13:03.161497 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153662 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:13:03.161497 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153665 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:13:03.161497 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153667 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:13:03.161497 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153670 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:13:03.161497 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153672 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:13:03.161497 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153675 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:13:03.161497 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153678 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:13:03.161497 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153680 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:13:03.161497 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153683 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:13:03.162086 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153686 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:13:03.162086 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153689 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:13:03.162086 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153692 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:13:03.162086 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153697 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:13:03.162086 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153700 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:13:03.162086 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153703 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:13:03.162086 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153706 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:13:03.162086 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153708 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:13:03.162086 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153711 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:13:03.162086 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153714 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:13:03.162086 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153716 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:13:03.162086 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153719 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:13:03.162086 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153722 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:13:03.162086 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153724 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:13:03.162086 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153727 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:13:03.162086 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153730 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:13:03.162086 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153732 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:13:03.162086 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153736 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:13:03.162086 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153739 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:13:03.162562 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153741 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:13:03.162562 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153744 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:13:03.162562 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153746 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:13:03.162562 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153749 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:13:03.162562 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153752 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:13:03.162562 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153754 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:13:03.162562 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153757 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:13:03.162562 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153759 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:13:03.162562 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153762 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:13:03.162562 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153764 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:13:03.162562 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.153790 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:13:03.162562 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153833 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:13:03.162562 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153838 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:13:03.162562 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153841 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:13:03.162562 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153844 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:13:03.162562 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153847 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:13:03.162948 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153850 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:13:03.162948 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153854 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:13:03.162948 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153857 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:13:03.162948 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153859 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:13:03.162948 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153862 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:13:03.162948 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153865 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:13:03.162948 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153867 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:13:03.162948 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153870 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:13:03.162948 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153873 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:13:03.162948 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153875 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:13:03.162948 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153878 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:13:03.162948 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153880 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:13:03.162948 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153883 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:13:03.162948 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153886 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:13:03.162948 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153890 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:13:03.162948 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153893 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:13:03.162948 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153896 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:13:03.162948 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153898 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:13:03.162948 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153901 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:13:03.162948 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153903 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:13:03.163427 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153906 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:13:03.163427 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153909 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:13:03.163427 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153911 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:13:03.163427 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153914 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:13:03.163427 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153916 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:13:03.163427 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153920 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:13:03.163427 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153923 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:13:03.163427 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153926 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:13:03.163427 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153929 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:13:03.163427 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153931 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:13:03.163427 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153934 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:13:03.163427 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153937 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:13:03.163427 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153940 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:13:03.163427 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153944 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:13:03.163427 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153947 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:13:03.163427 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153949 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:13:03.163427 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153952 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:13:03.163427 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153955 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:13:03.163427 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153957 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:13:03.163919 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153960 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:13:03.163919 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153963 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:13:03.163919 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153965 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:13:03.163919 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153968 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:13:03.163919 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153970 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:13:03.163919 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153973 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:13:03.163919 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153975 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:13:03.163919 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153979 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:13:03.163919 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153982 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:13:03.163919 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153985 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:13:03.163919 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153987 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:13:03.163919 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153990 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:13:03.163919 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153992 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:13:03.163919 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153995 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:13:03.163919 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.153997 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:13:03.163919 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154000 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:13:03.163919 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154002 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:13:03.163919 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154005 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:13:03.163919 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154007 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:13:03.163919 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154010 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:13:03.164386 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154015 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:13:03.164386 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154018 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:13:03.164386 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154022 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:13:03.164386 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154024 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:13:03.164386 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154027 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:13:03.164386 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154030 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:13:03.164386 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154034 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:13:03.164386 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154037 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:13:03.164386 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154040 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:13:03.164386 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154042 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:13:03.164386 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154045 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:13:03.164386 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154048 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:13:03.164386 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154050 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:13:03.164386 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154053 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:13:03.164386 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154055 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:13:03.164386 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154058 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:13:03.164386 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154060 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:13:03.164386 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154063 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:13:03.164386 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154065 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:13:03.164842 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154069 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:13:03.164842 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154072 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:13:03.164842 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154075 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:13:03.164842 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154077 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:13:03.164842 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154080 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:13:03.164842 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154083 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:13:03.164842 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154085 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:13:03.164842 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154088 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:13:03.164842 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154091 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:13:03.164842 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154094 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:13:03.164842 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154096 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:13:03.164842 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154098 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:13:03.164842 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154101 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:13:03.164842 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154103 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:13:03.164842 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154107 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:13:03.164842 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154110 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:13:03.164842 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154112 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:13:03.164842 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154114 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:13:03.164842 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154117 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:13:03.164842 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154121 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:13:03.165300 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154123 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:13:03.165300 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154126 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:13:03.165300 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.154128 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:13:03.165300 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.162573 2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:13:03.165300 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.162589 2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:13:03.165300 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.162612 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:13:03.165300 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.162618 2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:13:03.165300 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162674 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:13:03.165300 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162679 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:13:03.165300 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162682 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:13:03.165300 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162685 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:13:03.165300 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162690 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:13:03.165300 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162694 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:13:03.165300 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162697 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:13:03.165300 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162701 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:13:03.165300 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162704 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:13:03.165300 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162707 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:13:03.165300 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162709 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:13:03.165300 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162712 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:13:03.165756 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162715 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:13:03.165756 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162718 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:13:03.165756 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162720 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:13:03.165756 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162723 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:13:03.165756 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162726 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:13:03.165756 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162728 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:13:03.165756 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162732 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:13:03.165756 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162736 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:13:03.165756 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162739 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:13:03.165756 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162742 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:13:03.165756 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162744 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:13:03.165756 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162747 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:13:03.165756 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162750 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:13:03.165756 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162754 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:13:03.165756 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162756 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:13:03.165756 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162759 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:13:03.165756 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162762 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:13:03.165756 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162765 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:13:03.165756 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162767 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:13:03.166222 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162770 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:13:03.166222 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162773 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:13:03.166222 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162775 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:13:03.166222 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162778 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:13:03.166222 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162781 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:13:03.166222 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162783 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:13:03.166222 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162786 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:13:03.166222 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162789 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:13:03.166222 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162791 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:13:03.166222 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162794 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:13:03.166222 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162796 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:13:03.166222 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162799 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:13:03.166222 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162802 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:13:03.166222 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162804 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:13:03.166222 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162807 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:13:03.166222 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162815 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:13:03.166222 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162818 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:13:03.166222 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162820 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:13:03.166222 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162823 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:13:03.166222 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162825 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:13:03.166697 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162828 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:13:03.166697 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162831 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:13:03.166697 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162833 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:13:03.166697 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162836 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:13:03.166697 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162839 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:13:03.166697 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162841 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:13:03.166697 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162844 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:13:03.166697 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162847 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:13:03.166697 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162849 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:13:03.166697 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162852 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:13:03.166697 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162855 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:13:03.166697 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162858 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:13:03.166697 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162861 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:13:03.166697 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162863 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:13:03.166697 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162866 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:13:03.166697 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162868 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:13:03.166697 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162871 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:13:03.166697 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162874 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:13:03.166697 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162877 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:13:03.166697 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162879 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:13:03.167204 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162881 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:13:03.167204 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162884 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:13:03.167204 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162887 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:13:03.167204 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162889 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:13:03.167204 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162892 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:13:03.167204 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162894 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:13:03.167204 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162897 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:13:03.167204 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162899 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:13:03.167204 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162908 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:13:03.167204 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162911 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:13:03.167204 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162913 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:13:03.167204 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162916 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:13:03.167204 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162918 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:13:03.167204 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162921 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:13:03.167204 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162924 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:13:03.167204 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162926 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:13:03.167204 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162929 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:13:03.167204 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162931 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:13:03.167204 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162934 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:13:03.167666 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162937 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:13:03.167666 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162939 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:13:03.167666 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162942 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:13:03.167666 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162945 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:13:03.167666 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162947 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:13:03.167666 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162950 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:13:03.167666 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162952 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:13:03.167666 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162955 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:13:03.167666 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162958 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:13:03.167666 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162960 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:13:03.167666 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162963 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:13:03.167666 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162966 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:13:03.167666 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162969 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:13:03.167666 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162971 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:13:03.167666 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162974 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:13:03.167666 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.162976 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:13:03.168047 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.163004 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:13:03.168047 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163146 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:13:03.168047 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163152 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:13:03.168047 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163155 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:13:03.168047 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163158 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:13:03.168047 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163161 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:13:03.168047 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163164 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:13:03.168047 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163167 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:13:03.168047 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163170 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:13:03.168047 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163172 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:13:03.168047 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163175 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:13:03.168047 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163177 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:13:03.168047 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163180 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:13:03.168047 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163183 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:13:03.168047 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163185 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:13:03.168047 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163196 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:13:03.168415 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163199 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:13:03.168415 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163202 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:13:03.168415 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163205 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:13:03.168415 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163207 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:13:03.168415 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163210 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:13:03.168415 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163212 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:13:03.168415 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163215 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:13:03.168415 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163218 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:13:03.168415 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163220 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:13:03.168415 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163223 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:13:03.168415 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163225 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:13:03.168415 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163228 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:13:03.168415 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163231 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:13:03.168415 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163233 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:13:03.168415 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163236 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:13:03.168415 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163239 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:13:03.168415 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163241 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:13:03.168415 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163243 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:13:03.168415 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163246 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:13:03.168899 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163249 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:13:03.168899 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163253 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:13:03.168899 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163257 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:13:03.168899 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163260 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:13:03.168899 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163265 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:13:03.168899 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163268 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:13:03.168899 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163271 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:13:03.168899 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163273 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:13:03.168899 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163276 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:13:03.168899 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163278 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:13:03.168899 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163281 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:13:03.168899 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163283 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:13:03.168899 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163286 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:13:03.168899 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163289 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:13:03.168899 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163291 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:13:03.168899 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163294 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:13:03.168899 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163296 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:13:03.168899 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163299 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:13:03.169306 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163301 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:13:03.169306 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163304 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:13:03.169306 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163306 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:13:03.169306 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163309 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:13:03.169306 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163311 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:13:03.169306 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163314 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:13:03.169306 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163317 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:13:03.169306 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163319 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:13:03.169306 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163322 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:13:03.169306 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163324 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:13:03.169306 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163327 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:13:03.169306 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163329 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:13:03.169306 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163332 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:13:03.169306 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163334 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:13:03.169306 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163337 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:13:03.169306 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163340 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:13:03.169306 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163342 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:13:03.169306 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163345 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:13:03.169306 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163347 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:13:03.169306 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163350 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:13:03.169797 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163353 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:13:03.169797 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163355 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:13:03.169797 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163358 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:13:03.169797 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163360 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:13:03.169797 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163362 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:13:03.169797 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163366 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:13:03.169797 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163370 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:13:03.169797 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163373 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:13:03.169797 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163377 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:13:03.169797 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163379 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:13:03.169797 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163382 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:13:03.169797 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163384 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:13:03.169797 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163387 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:13:03.169797 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163390 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:13:03.169797 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163393 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:13:03.169797 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163395 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:13:03.169797 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163398 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:13:03.169797 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163405 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:13:03.169797 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163407 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:13:03.170234 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163410 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:13:03.170234 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163413 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:13:03.170234 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163415 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:13:03.170234 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163418 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:13:03.170234 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163420 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:13:03.170234 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163423 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:13:03.170234 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163425 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:13:03.170234 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163428 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:13:03.170234 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163431 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:13:03.170234 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163434 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:13:03.170234 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163437 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:13:03.170234 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163439 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:13:03.170234 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163442 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:13:03.170234 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163445 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:13:03.170234 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163462 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:13:03.170234 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.163487 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:13:03.170674 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163520 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:13:03.170674 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163523 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:13:03.170674 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163526 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:13:03.170674 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163529 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:13:03.170674 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163532 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:13:03.170674 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163534 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:13:03.170674 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163537 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:13:03.170674 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163540 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:13:03.170674 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163543 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:13:03.170674 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163546 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:13:03.170674 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163548 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:13:03.170674 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163551 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:13:03.170674 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163553 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:13:03.170674 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163556 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:13:03.170674 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163559 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:13:03.170674 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163562 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:13:03.170674 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163565 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:13:03.170674 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163567 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:13:03.170674 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163570 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:13:03.170674 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163573 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:13:03.171151 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163575 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:13:03.171151 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163578 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:13:03.171151 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163581 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:13:03.171151 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163583 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:13:03.171151 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163586 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:13:03.171151 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163588 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:13:03.171151 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163591 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:13:03.171151 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163593 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:13:03.171151 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163596 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:13:03.171151 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163598 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:13:03.171151 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163602 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:13:03.171151 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163604 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:13:03.171151 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163607 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:13:03.171151 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163609 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:13:03.171151 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163612 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:13:03.171151 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163615 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:13:03.171151 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163617 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:13:03.171151 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163619 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:13:03.171151 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163622 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:13:03.171151 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163624 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:13:03.171636 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163627 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:13:03.171636 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163629 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:13:03.171636 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163633 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:13:03.171636 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163636 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:13:03.171636 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163639 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:13:03.171636 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163642 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:13:03.171636 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163645 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:13:03.171636 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163647 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:13:03.171636 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163650 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:13:03.171636 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163652 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:13:03.171636 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163655 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:13:03.171636 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163657 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:13:03.171636 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163660 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:13:03.171636 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163662 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:13:03.171636 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163665 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:13:03.171636 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163668 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:13:03.171636 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163670 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:13:03.171636 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163673 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:13:03.172047 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163675 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:13:03.172047 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163678 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:13:03.172047 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163680 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:13:03.172047 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163683 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:13:03.172047 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163686 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:13:03.172047 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163689 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:13:03.172047 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163692 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:13:03.172047 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163694 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:13:03.172047 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163697 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:13:03.172047 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163699 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:13:03.172047 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163701 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:13:03.172047 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163704 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:13:03.172047 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163707 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:13:03.172047 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163709 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:13:03.172047 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163712 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:13:03.172047 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163714 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:13:03.172047 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163717 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:13:03.172047 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163719 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:13:03.172047 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163724 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:13:03.172047 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163726 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:13:03.172591 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163729 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:13:03.172591 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163731 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:13:03.172591 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163734 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:13:03.172591 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163736 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:13:03.172591 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163739 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:13:03.172591 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163742 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:13:03.172591 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163745 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:13:03.172591 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163748 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:13:03.172591 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163751 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:13:03.172591 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163754 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:13:03.172591 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163757 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:13:03.172591 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163759 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:13:03.172591 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163762 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:13:03.172591 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163764 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:13:03.172591 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163767 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:13:03.172591 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163771 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:13:03.172591 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163773 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:13:03.172591 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163776 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:13:03.172591 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163779 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:13:03.173046 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163782 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:13:03.173046 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163784 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:13:03.173046 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163787 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:13:03.173046 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163789 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:13:03.173046 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163792 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:13:03.173046 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163794 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:13:03.173046 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163797 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:13:03.173046 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163799 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:13:03.173046 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:03.163802 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:13:03.173046 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.163852 2577 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:13:03.173046 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.165968 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 21:13:03.173046 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.167013 2577 server.go:1014] "Starting client certificate rotation" Oct 30 21:13:03.173046 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.167122 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:13:03.173046 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.167160 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:13:03.173046 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.169946 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:13:03.173046 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.171138 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:13:03.180761 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.180747 2577 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:13:03.184609 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.184594 2577 log.go:25] "Validated CRI v1 image API" Oct 30 21:13:03.185443 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.185412 2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:13:03.188507 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.188487 2577 fs.go:135] Filesystem UUIDs: map[4a4bd9b2-9de1-493b-b9de-fa09a6e7f48f:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d02f50c3-c68e-4a70-acb7-cdc6206c5992:/dev/nvme0n1p4] Oct 30 21:13:03.188579 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.188506 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 21:13:03.193816 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.193708 2577 manager.go:217] Machine: {Timestamp:2026-10-30 21:13:03.193237234 +0000 UTC m=+0.371120195 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099236 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec291bed37d7e4f77b7ff30252ec9ef6 SystemUUID:ec291bed-37d7-e4f7-7b7f-f30252ec9ef6 BootID:7d0ed3b0-8d3a-4a2d-b3aa-bffd15478260 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:d5:e2:59:bb Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:d5:e2:59:bb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:24:2f:c0:3c:2c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:13:03.194613 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.194600 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:13:03.194693 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.194680 2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:13:03.197425 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.197402 2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:13:03.197597 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.197424 2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-83.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:13:03.197662 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.197601 2577 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:13:03.197662 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.197609 2577 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:13:03.197662 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.197619 2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:13:03.197662 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.197631 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:13:03.198630 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.198619 2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:13:03.202408 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.202393 2577 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:13:03.202552 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.202541 2577 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:13:03.203356 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.203346 2577 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:13:03.203400 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.203362 2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:13:03.203400 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.203373 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:13:03.203400 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.203382 2577 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:13:03.203400 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.203391 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:13:03.203791 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.203773 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:13:03.207486 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.207465 2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:13:03.213581 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.213564 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:13:03.214437 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.214423 2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:13:03.214529 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.214479 2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:13:03.214703 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.214690 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:13:03.214765 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.214709 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:13:03.214765 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.214718 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:13:03.214765 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.214727 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:13:03.214765 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.214736 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:13:03.214765 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.214746 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:13:03.215485 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.215474 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:13:03.215544 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.215489 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:13:03.215544 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.215501 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:13:03.215544 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.215511 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:13:03.215544 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.215525 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:13:03.215544 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.215539 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:13:03.216966 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.216954 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:13:03.217033 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.216970 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:13:03.220873 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.220857 2577 server.go:1263] "Started kubelet" Oct 30 21:13:03.220962 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.220916 2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:13:03.221076 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.221025 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:13:03.221126 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.221102 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:13:03.222171 ip-10-0-129-83 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:13:03.222606 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.222582 2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:13:03.224272 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.224259 2577 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:13:03.228277 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.228257 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:13:03.229528 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.229511 2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:13:03.229528 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:03.229520 2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 21:13:03.230297 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.230269 2577 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:13:03.230297 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.230286 2577 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:13:03.230784 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.230399 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:13:03.230784 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.230498 2577 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:13:03.230784 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.230506 2577 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:13:03.231444 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:03.231394 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-83.us-west-2.compute.internal\" not found" Oct 30 21:13:03.232086 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.232065 2577 factory.go:55] Registering systemd factory Oct 30 21:13:03.232086 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.232088 2577 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:13:03.232313 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.232300 2577 factory.go:153] Registering CRI-O factory Oct 30 21:13:03.232355 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.232316 2577 factory.go:223] Registration of the crio container factory successfully Oct 30 21:13:03.232383 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.232362 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:13:03.232411 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.232384 2577 factory.go:103] Registering Raw factory Oct 30 21:13:03.232439 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.232417 2577 manager.go:1199] Started watching for new ooms in manager Oct 30 21:13:03.233376 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.233364 2577 manager.go:319] Starting recovery of all containers Oct 30 21:13:03.243795 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.243776 2577 manager.go:324] Recovery completed Oct 30 21:13:03.248675 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.248658 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:13:03.250209 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.250191 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:13:03.250270 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.250220 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:13:03.250270 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.250231 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:13:03.252030 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.250955 2577 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:13:03.252030 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.250984 2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:13:03.252030 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.251020 2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:13:03.252030 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.251518 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:13:03.252491 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.252469 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:13:03.252569 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.252503 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:13:03.252569 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.252516 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:13:03.252569 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.252540 2577 policy_none.go:50] "Start" Oct 30 21:13:03.252569 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.252554 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:13:03.252569 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.252564 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:13:03.253729 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.253719 2577 policy_none.go:44] "Start" Oct 30 21:13:03.304215 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.304052 2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:13:03.308799 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.308782 2577 manager.go:341] "Starting Device Plugin manager" Oct 30 21:13:03.309461 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:03.309430 2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:13:03.309547 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.309469 2577 server.go:87] "Starting device plugin registration server" Oct 30 21:13:03.309738 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.309729 2577 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:13:03.309817 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.309743 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:13:03.309868 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.309856 2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:13:03.309953 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.309940 2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:13:03.310005 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.309953 2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:13:03.310507 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:03.310481 2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:13:03.310586 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:03.310537 2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-83.us-west-2.compute.internal\" not found" Oct 30 21:13:03.359225 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:03.357168 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-83.us-west-2.compute.internal.18e36b0db3ea88aa default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-83.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-83.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 21:13:03.22082833 +0000 UTC m=+0.398711292,LastTimestamp:2026-10-30 21:13:03.22082833 +0000 UTC m=+0.398711292,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-83.us-west-2.compute.internal,}" Oct 30 21:13:03.374232 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.374203 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:13:03.375483 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.375466 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:13:03.375547 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.375492 2577 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:13:03.375547 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.375513 2577 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:13:03.375644 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:03.375553 2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 21:13:03.402787 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.402719 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-83.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 21:13:03.402787 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:03.402738 2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-83.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 21:13:03.411139 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.411125 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:13:03.412434 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.412417 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:13:03.412526 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.412464 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:13:03.412526 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.412478 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:13:03.412526 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.412501 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:03.422354 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:03.422331 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 21:13:03.422354 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:03.422332 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 21:13:03.422498 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:03.422414 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-83.us-west-2.compute.internal.18e36b0db5aad26b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-83.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-83.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-83.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 21:13:03.250207339 +0000 UTC m=+0.428090304,LastTimestamp:2026-10-30 21:13:03.250207339 +0000 UTC m=+0.428090304,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-83.us-west-2.compute.internal,}" Oct 30 21:13:03.436171 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.436150 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:13:03.436244 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.436188 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6k65j" Oct 30 21:13:03.476369 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.476343 2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-83.us-west-2.compute.internal"] Oct 30 21:13:03.476463 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.476402 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:13:03.477287 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.477272 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:13:03.477364 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.477298 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:13:03.477364 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.477309 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:13:03.479432 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.479421 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:13:03.479578 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.479565 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:03.479612 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.479594 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:13:03.480191 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.480171 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:13:03.480271 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.480209 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:13:03.480271 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.480223 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:13:03.480271 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.480226 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:13:03.480271 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.480251 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:13:03.480271 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.480268 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:13:03.482299 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.482285 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:03.482361 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.482309 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:13:03.482972 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.482957 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:13:03.483060 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.482988 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:13:03.483060 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.483001 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:13:03.511628 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.511609 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:13:03.512537 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.512519 2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:13:03.514078 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:03.514061 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-83.us-west-2.compute.internal\" not found" node="ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:03.516870 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:03.516854 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-83.us-west-2.compute.internal\" not found" node="ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:03.532900 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.532883 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e151e7c539d69d81e686cdbfb721e69-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal\" (UID: \"2e151e7c539d69d81e686cdbfb721e69\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:03.533008 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.532908 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e151e7c539d69d81e686cdbfb721e69-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal\" (UID: \"2e151e7c539d69d81e686cdbfb721e69\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:03.533008 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.532925 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b9addd42169977c49b17223d2454db0-config\") pod \"kube-apiserver-proxy-ip-10-0-129-83.us-west-2.compute.internal\" (UID: \"8b9addd42169977c49b17223d2454db0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:03.557675 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.557658 2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:03.557728 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:03.557677 2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-83.us-west-2.compute.internal\": node \"ip-10-0-129-83.us-west-2.compute.internal\" not found" Oct 30 21:13:03.631887 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.631865 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:03.633752 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.633735 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e151e7c539d69d81e686cdbfb721e69-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal\" (UID: \"2e151e7c539d69d81e686cdbfb721e69\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:03.633809 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.633760 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b9addd42169977c49b17223d2454db0-config\") pod \"kube-apiserver-proxy-ip-10-0-129-83.us-west-2.compute.internal\" (UID: \"8b9addd42169977c49b17223d2454db0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:03.633809 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.633784 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e151e7c539d69d81e686cdbfb721e69-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal\" (UID: \"2e151e7c539d69d81e686cdbfb721e69\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:03.633871 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.633809 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e151e7c539d69d81e686cdbfb721e69-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal\" (UID: \"2e151e7c539d69d81e686cdbfb721e69\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:03.633871 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.633839 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e151e7c539d69d81e686cdbfb721e69-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal\" (UID: \"2e151e7c539d69d81e686cdbfb721e69\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:03.633871 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.633839 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b9addd42169977c49b17223d2454db0-config\") pod \"kube-apiserver-proxy-ip-10-0-129-83.us-west-2.compute.internal\" (UID: \"8b9addd42169977c49b17223d2454db0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:03.687183 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.687128 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:13:03.688402 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:03.688391 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:05.487857 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:05.487839 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:13:05.558900 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:05.558868 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:05.562020 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:05.562005 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:05.908157 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:05.908084 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 21:13:05.908653 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:05.908241 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 21:13:05.908653 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:05.908295 2577 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://aa9979ce62f3e4f9a86916a6635549a1-220db26bc9a348ee.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-129-83.us-west-2.compute.internal.18e36b0db5ab3da8\": read tcp 10.0.129.83:42840->18.226.184.238:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-129-83.us-west-2.compute.internal.18e36b0db5ab3da8 default 5034 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-83.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-83.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-83.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 21:13:03 +0000 UTC,LastTimestamp:2026-10-30 21:13:03.477312671 +0000 UTC m=+0.655195633,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-83.us-west-2.compute.internal,}" Oct 30 21:13:05.944708 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:05.944676 2577 apiserver.go:52] "Watching apiserver" Oct 30 21:13:05.969446 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:05.969425 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 21:13:06.181832 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.181748 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:08:05 +0000 UTC" deadline="2028-06-01 11:45:44.802370167 +0000 UTC" Oct 30 21:13:06.181832 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.181783 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13910h32m38.620589439s" Oct 30 21:13:06.227354 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.227313 2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:13:06.227753 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.227712 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-jztdg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2","openshift-cluster-node-tuning-operator/tuned-pgnjv","openshift-multus/multus-additional-cni-plugins-brt6h","kube-system/konnectivity-agent-vpxpj","openshift-dns/node-resolver-p4c9m","openshift-ovn-kubernetes/ovnkube-node-jbjqt","openshift-multus/multus-zn2wp","openshift-network-operator/iptables-alerter-jfprx","kube-system/kube-apiserver-proxy-ip-10-0-129-83.us-west-2.compute.internal","openshift-image-registry/node-ca-q8g7b","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-n4g68","openshift-network-diagnostics/network-check-target-lw64d"] Oct 30 21:13:06.229375 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.229357 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:13:06.230224 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.230209 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:06.230303 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:06.230263 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jztdg" podUID="3341bf47-8c8f-4073-8758-122e04e0b138" Oct 30 21:13:06.232234 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.232217 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:06.234290 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.234273 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.236402 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.236385 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:06.238404 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.238388 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vpxpj" Oct 30 21:13:06.240499 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.240483 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p4c9m" Oct 30 21:13:06.242639 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.242624 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.244873 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.244848 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.246983 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.246966 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jfprx" Oct 30 21:13:06.249226 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.249204 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q8g7b" Oct 30 21:13:06.251485 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.251468 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4g68" Oct 30 21:13:06.251573 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:06.251519 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4g68" podUID="30bde564-b785-4359-9377-ffec880fb18c" Oct 30 21:13:06.253694 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.253677 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lw64d" Oct 30 21:13:06.253779 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:06.253740 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lw64d" podUID="19538cda-3ba8-4bad-bb47-23a2511736e7" Oct 30 21:13:06.272578 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.272561 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:13:06.281523 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.281499 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-kubernetes\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.281599 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.281531 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/08f1ac32-f49e-4aa2-9ab8-8a1aac7bef36-konnectivity-ca\") pod \"konnectivity-agent-vpxpj\" (UID: \"08f1ac32-f49e-4aa2-9ab8-8a1aac7bef36\") " pod="kube-system/konnectivity-agent-vpxpj" Oct 30 21:13:06.281599 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.281547 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-multus-conf-dir\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.281599 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.281561 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-run-netns\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.281599 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.281582 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-var-lib-openvswitch\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.281737 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.281625 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-log-socket\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.281737 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.281675 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-run-ovn-kubernetes\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.281737 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.281705 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-sysconfig\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.281737 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.281721 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25ca3b5e-7eba-4411-be09-5da0fe41e473-system-cni-dir\") pod \"multus-additional-cni-plugins-brt6h\" (UID: \"25ca3b5e-7eba-4411-be09-5da0fe41e473\") " pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:06.281863 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.281758 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-host-var-lib-cni-bin\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.281863 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.281772 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-host-run-multus-certs\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.281863 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.281786 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsrnm\" (UniqueName: \"kubernetes.io/projected/19538cda-3ba8-4bad-bb47-23a2511736e7-kube-api-access-bsrnm\") pod \"network-check-target-lw64d\" (UID: \"19538cda-3ba8-4bad-bb47-23a2511736e7\") " pod="openshift-network-diagnostics/network-check-target-lw64d" Oct 30 21:13:06.281863 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.281802 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtvrj\" (UniqueName: \"kubernetes.io/projected/9597db44-1ca0-4fa8-ade7-cb55eeca9593-kube-api-access-wtvrj\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:06.281863 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.281823 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-lib-modules\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.282018 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.281874 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-tuned\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.282018 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.281894 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-cni-binary-copy\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.282018 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.281909 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54b8c0a5-e1a7-4d02-a402-b3cadc139e28-serviceca\") pod \"node-ca-q8g7b\" (UID: \"54b8c0a5-e1a7-4d02-a402-b3cadc139e28\") " pod="openshift-image-registry/node-ca-q8g7b" Oct 30 21:13:06.282018 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.281923 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-kubelet\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.282018 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.281937 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-sysctl-conf\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.282018 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.282002 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/25ca3b5e-7eba-4411-be09-5da0fe41e473-cnibin\") pod \"multus-additional-cni-plugins-brt6h\" (UID: \"25ca3b5e-7eba-4411-be09-5da0fe41e473\") " pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:06.282199 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.282020 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-etc-kubernetes\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.282199 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.282059 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9597db44-1ca0-4fa8-ade7-cb55eeca9593-socket-dir\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:06.282199 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.282083 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30bde564-b785-4359-9377-ffec880fb18c-metrics-certs\") pod \"network-metrics-daemon-n4g68\" (UID: \"30bde564-b785-4359-9377-ffec880fb18c\") " pod="openshift-multus/network-metrics-daemon-n4g68" Oct 30 21:13:06.282199 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.282098 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/25ca3b5e-7eba-4411-be09-5da0fe41e473-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-brt6h\" (UID: \"25ca3b5e-7eba-4411-be09-5da0fe41e473\") " pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:06.282199 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.282132 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-multus-cni-dir\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.282199 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.282157 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3341bf47-8c8f-4073-8758-122e04e0b138-dbus\") pod \"global-pull-secret-syncer-jztdg\" (UID: \"3341bf47-8c8f-4073-8758-122e04e0b138\") " pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:06.282199 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.282172 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-cni-netd\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.282430 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.282216 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cctxz\" (UniqueName: \"kubernetes.io/projected/05d96464-f946-4a83-88e5-ececfee30b4c-kube-api-access-cctxz\") pod \"node-resolver-p4c9m\" (UID: \"05d96464-f946-4a83-88e5-ececfee30b4c\") " pod="openshift-dns/node-resolver-p4c9m" Oct 30 21:13:06.282430 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.282237 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-multus-daemon-config\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.282430 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.282251 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54b8c0a5-e1a7-4d02-a402-b3cadc139e28-host\") pod \"node-ca-q8g7b\" (UID: \"54b8c0a5-e1a7-4d02-a402-b3cadc139e28\") " pod="openshift-image-registry/node-ca-q8g7b" Oct 30 21:13:06.282430 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.282293 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/18d4542b-53c1-473f-9605-c82fd5a642f4-ovnkube-config\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.282430 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.282318 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-sysctl-d\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.282430 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.282334 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05d96464-f946-4a83-88e5-ececfee30b4c-tmp-dir\") pod \"node-resolver-p4c9m\" (UID: \"05d96464-f946-4a83-88e5-ececfee30b4c\") " pod="openshift-dns/node-resolver-p4c9m" Oct 30 21:13:06.282430 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.282380 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-host-run-k8s-cni-cncf-io\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.282430 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.282404 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg66z\" (UniqueName: \"kubernetes.io/projected/54b8c0a5-e1a7-4d02-a402-b3cadc139e28-kube-api-access-jg66z\") pod \"node-ca-q8g7b\" (UID: \"54b8c0a5-e1a7-4d02-a402-b3cadc139e28\") " pod="openshift-image-registry/node-ca-q8g7b" Oct 30 21:13:06.282430 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.282424 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-slash\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.282700 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.282439 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-run-systemd\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.282700 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.282463 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvgz2\" (UniqueName: \"kubernetes.io/projected/1d4aae53-afc9-4373-953d-b9b7c323e3ee-kube-api-access-tvgz2\") pod \"iptables-alerter-jfprx\" (UID: \"1d4aae53-afc9-4373-953d-b9b7c323e3ee\") " pod="openshift-network-operator/iptables-alerter-jfprx" Oct 30 21:13:06.282700 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.282483 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/25ca3b5e-7eba-4411-be09-5da0fe41e473-cni-binary-copy\") pod \"multus-additional-cni-plugins-brt6h\" (UID: \"25ca3b5e-7eba-4411-be09-5da0fe41e473\") " pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:06.282700 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.282498 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/25ca3b5e-7eba-4411-be09-5da0fe41e473-tuning-conf-dir\") pod \"multus-additional-cni-plugins-brt6h\" (UID: \"25ca3b5e-7eba-4411-be09-5da0fe41e473\") " pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:06.282700 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.282520 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1d4aae53-afc9-4373-953d-b9b7c323e3ee-iptables-alerter-script\") pod \"iptables-alerter-jfprx\" (UID: \"1d4aae53-afc9-4373-953d-b9b7c323e3ee\") " pod="openshift-network-operator/iptables-alerter-jfprx" Oct 30 21:13:06.282700 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.282547 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl9th\" (UniqueName: \"kubernetes.io/projected/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-kube-api-access-hl9th\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.282700 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.282569 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-hostroot\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.282700 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.282630 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-run\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.282700 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.282657 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-var-lib-kubelet\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.282700 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.282672 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-tmp\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.282700 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.282685 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/05d96464-f946-4a83-88e5-ececfee30b4c-hosts-file\") pod \"node-resolver-p4c9m\" (UID: \"05d96464-f946-4a83-88e5-ececfee30b4c\") " pod="openshift-dns/node-resolver-p4c9m" Oct 30 21:13:06.283034 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.282707 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-multus-socket-dir-parent\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.283034 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.282739 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-host-var-lib-kubelet\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.283034 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.282757 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-modprobe-d\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.283034 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.282780 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/25ca3b5e-7eba-4411-be09-5da0fe41e473-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-brt6h\" (UID: \"25ca3b5e-7eba-4411-be09-5da0fe41e473\") " pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:06.283034 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.282808 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3341bf47-8c8f-4073-8758-122e04e0b138-kubelet-config\") pod \"global-pull-secret-syncer-jztdg\" (UID: \"3341bf47-8c8f-4073-8758-122e04e0b138\") " pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:06.283034 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.282825 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9597db44-1ca0-4fa8-ade7-cb55eeca9593-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:06.283034 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.282848 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/18d4542b-53c1-473f-9605-c82fd5a642f4-ovnkube-script-lib\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.283034 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.282874 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k52xf\" (UniqueName: \"kubernetes.io/projected/18d4542b-53c1-473f-9605-c82fd5a642f4-kube-api-access-k52xf\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.283388 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.283137 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d4aae53-afc9-4373-953d-b9b7c323e3ee-host-slash\") pod \"iptables-alerter-jfprx\" (UID: \"1d4aae53-afc9-4373-953d-b9b7c323e3ee\") " pod="openshift-network-operator/iptables-alerter-jfprx" Oct 30 21:13:06.283388 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.283211 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-systemd\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.283388 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.283255 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-run-openvswitch\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.283388 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.283334 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-node-log\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.283568 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.283398 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.283568 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.283434 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-cnibin\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.283568 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.283486 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-cni-bin\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.283568 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.283521 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8884\" (UniqueName: \"kubernetes.io/projected/30bde564-b785-4359-9377-ffec880fb18c-kube-api-access-l8884\") pod \"network-metrics-daemon-n4g68\" (UID: \"30bde564-b785-4359-9377-ffec880fb18c\") " pod="openshift-multus/network-metrics-daemon-n4g68" Oct 30 21:13:06.283731 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.283571 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-host-var-lib-cni-multus\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.283731 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.283604 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3341bf47-8c8f-4073-8758-122e04e0b138-original-pull-secret\") pod \"global-pull-secret-syncer-jztdg\" (UID: \"3341bf47-8c8f-4073-8758-122e04e0b138\") " pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:06.283731 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.283633 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9597db44-1ca0-4fa8-ade7-cb55eeca9593-registration-dir\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:06.283731 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.283656 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9597db44-1ca0-4fa8-ade7-cb55eeca9593-device-dir\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:06.283731 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.283685 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9597db44-1ca0-4fa8-ade7-cb55eeca9593-sys-fs\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:06.283731 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.283715 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-sys\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.283979 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.283785 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-host\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.283979 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.283814 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/08f1ac32-f49e-4aa2-9ab8-8a1aac7bef36-agent-certs\") pod \"konnectivity-agent-vpxpj\" (UID: \"08f1ac32-f49e-4aa2-9ab8-8a1aac7bef36\") " pod="kube-system/konnectivity-agent-vpxpj" Oct 30 21:13:06.283979 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.283843 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9597db44-1ca0-4fa8-ade7-cb55eeca9593-etc-selinux\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:06.283979 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.283868 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-systemd-units\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.283979 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.283898 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-etc-openvswitch\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.283979 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.283924 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-run-ovn\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.283979 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.283950 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/18d4542b-53c1-473f-9605-c82fd5a642f4-env-overrides\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.283979 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.283977 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvbqm\" (UniqueName: \"kubernetes.io/projected/25ca3b5e-7eba-4411-be09-5da0fe41e473-kube-api-access-mvbqm\") pod \"multus-additional-cni-plugins-brt6h\" (UID: \"25ca3b5e-7eba-4411-be09-5da0fe41e473\") " pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:06.284325 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.284001 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-system-cni-dir\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.284325 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.284027 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-host-run-netns\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.284325 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.284054 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8snd\" (UniqueName: \"kubernetes.io/projected/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-kube-api-access-k8snd\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.284325 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.284090 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/18d4542b-53c1-473f-9605-c82fd5a642f4-ovn-node-metrics-cert\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.286835 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.286818 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:13:06.286963 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.286952 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:13:06.287022 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.286961 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:13:06.287022 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.286961 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:13:06.287656 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.287642 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-x6zwr\"" Oct 30 21:13:06.287656 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.287645 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ph6zc\"" Oct 30 21:13:06.287884 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.287872 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:13:06.289104 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.289089 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:13:06.291853 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.291840 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qzjzq\"" Oct 30 21:13:06.292439 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.292424 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-dpfbq\"" Oct 30 21:13:06.292664 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.292651 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:13:06.292868 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.292844 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:13:06.292945 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.292899 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:13:06.292945 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.292936 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:13:06.293010 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.292979 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:13:06.293010 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.292998 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:13:06.293720 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.293710 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:13:06.332319 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.332298 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kvf2h\"" Oct 30 21:13:06.381405 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.381384 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:13:06.381488 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.381447 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rxk65" Oct 30 21:13:06.384530 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.384510 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-tmp\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.384624 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.384539 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/05d96464-f946-4a83-88e5-ececfee30b4c-hosts-file\") pod \"node-resolver-p4c9m\" (UID: \"05d96464-f946-4a83-88e5-ececfee30b4c\") " pod="openshift-dns/node-resolver-p4c9m" Oct 30 21:13:06.384624 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.384561 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-multus-socket-dir-parent\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.384624 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.384578 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-host-var-lib-kubelet\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.384624 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.384599 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-modprobe-d\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.384815 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.384622 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/25ca3b5e-7eba-4411-be09-5da0fe41e473-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-brt6h\" (UID: \"25ca3b5e-7eba-4411-be09-5da0fe41e473\") " pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:06.384815 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.384649 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3341bf47-8c8f-4073-8758-122e04e0b138-kubelet-config\") pod \"global-pull-secret-syncer-jztdg\" (UID: \"3341bf47-8c8f-4073-8758-122e04e0b138\") " pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:06.384815 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.384653 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-host-var-lib-kubelet\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.384815 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.384670 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9597db44-1ca0-4fa8-ade7-cb55eeca9593-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:06.384815 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.384705 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9597db44-1ca0-4fa8-ade7-cb55eeca9593-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:06.384815 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.384706 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/18d4542b-53c1-473f-9605-c82fd5a642f4-ovnkube-script-lib\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.384815 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.384657 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/05d96464-f946-4a83-88e5-ececfee30b4c-hosts-file\") pod \"node-resolver-p4c9m\" (UID: \"05d96464-f946-4a83-88e5-ececfee30b4c\") " pod="openshift-dns/node-resolver-p4c9m" Oct 30 21:13:06.384815 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.384726 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-multus-socket-dir-parent\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.384815 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.384736 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k52xf\" (UniqueName: \"kubernetes.io/projected/18d4542b-53c1-473f-9605-c82fd5a642f4-kube-api-access-k52xf\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.384815 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.384762 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3341bf47-8c8f-4073-8758-122e04e0b138-kubelet-config\") pod \"global-pull-secret-syncer-jztdg\" (UID: \"3341bf47-8c8f-4073-8758-122e04e0b138\") " pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:06.384815 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.384792 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d4aae53-afc9-4373-953d-b9b7c323e3ee-host-slash\") pod \"iptables-alerter-jfprx\" (UID: \"1d4aae53-afc9-4373-953d-b9b7c323e3ee\") " pod="openshift-network-operator/iptables-alerter-jfprx" Oct 30 21:13:06.385352 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.384823 2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:13:06.385352 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.384838 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-systemd\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.385352 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.384859 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-run-openvswitch\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.385352 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.384896 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-node-log\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.385352 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.384901 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d4aae53-afc9-4373-953d-b9b7c323e3ee-host-slash\") pod \"iptables-alerter-jfprx\" (UID: \"1d4aae53-afc9-4373-953d-b9b7c323e3ee\") " pod="openshift-network-operator/iptables-alerter-jfprx" Oct 30 21:13:06.385352 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.384924 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.385352 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.384940 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-run-openvswitch\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.385352 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.384926 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-systemd\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.385352 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.384975 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.385352 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.384987 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-node-log\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.385352 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385014 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-cnibin\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.385352 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385045 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-cni-bin\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.385352 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385080 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-cnibin\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.385352 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385088 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8884\" (UniqueName: \"kubernetes.io/projected/30bde564-b785-4359-9377-ffec880fb18c-kube-api-access-l8884\") pod \"network-metrics-daemon-n4g68\" (UID: \"30bde564-b785-4359-9377-ffec880fb18c\") " pod="openshift-multus/network-metrics-daemon-n4g68" Oct 30 21:13:06.385352 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385122 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-host-var-lib-cni-multus\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.385352 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385139 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-cni-bin\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.385352 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385121 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-modprobe-d\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.385352 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385149 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3341bf47-8c8f-4073-8758-122e04e0b138-original-pull-secret\") pod \"global-pull-secret-syncer-jztdg\" (UID: \"3341bf47-8c8f-4073-8758-122e04e0b138\") " pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:06.386234 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385160 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-host-var-lib-cni-multus\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.386234 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385176 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/25ca3b5e-7eba-4411-be09-5da0fe41e473-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-brt6h\" (UID: \"25ca3b5e-7eba-4411-be09-5da0fe41e473\") " pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:06.386234 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385181 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9597db44-1ca0-4fa8-ade7-cb55eeca9593-registration-dir\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:06.386234 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385209 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9597db44-1ca0-4fa8-ade7-cb55eeca9593-device-dir\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:06.386234 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:06.385223 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:13:06.386234 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385232 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9597db44-1ca0-4fa8-ade7-cb55eeca9593-sys-fs\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:06.386234 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385253 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-sys\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.386234 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:06.385278 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3341bf47-8c8f-4073-8758-122e04e0b138-original-pull-secret podName:3341bf47-8c8f-4073-8758-122e04e0b138 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:06.885259528 +0000 UTC m=+2.322254070 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3341bf47-8c8f-4073-8758-122e04e0b138-original-pull-secret") pod "global-pull-secret-syncer-jztdg" (UID: "3341bf47-8c8f-4073-8758-122e04e0b138") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:13:06.386234 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385289 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9597db44-1ca0-4fa8-ade7-cb55eeca9593-device-dir\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:06.386234 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385305 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-host\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.386234 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385309 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-sys\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.386234 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385276 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9597db44-1ca0-4fa8-ade7-cb55eeca9593-registration-dir\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:06.386234 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385324 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9597db44-1ca0-4fa8-ade7-cb55eeca9593-sys-fs\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:06.386234 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385331 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/08f1ac32-f49e-4aa2-9ab8-8a1aac7bef36-agent-certs\") pod \"konnectivity-agent-vpxpj\" (UID: \"08f1ac32-f49e-4aa2-9ab8-8a1aac7bef36\") " pod="kube-system/konnectivity-agent-vpxpj" Oct 30 21:13:06.386234 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385375 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-host\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.386234 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385401 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9597db44-1ca0-4fa8-ade7-cb55eeca9593-etc-selinux\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:06.386234 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385424 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-systemd-units\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.387026 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385446 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-etc-openvswitch\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.387026 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385455 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-systemd-units\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.387026 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385469 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-run-ovn\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.387026 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385492 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9597db44-1ca0-4fa8-ade7-cb55eeca9593-etc-selinux\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:06.387026 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385501 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/18d4542b-53c1-473f-9605-c82fd5a642f4-env-overrides\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.387026 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385506 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-run-ovn\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.387026 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385528 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvbqm\" (UniqueName: \"kubernetes.io/projected/25ca3b5e-7eba-4411-be09-5da0fe41e473-kube-api-access-mvbqm\") pod \"multus-additional-cni-plugins-brt6h\" (UID: \"25ca3b5e-7eba-4411-be09-5da0fe41e473\") " pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:06.387026 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385535 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-etc-openvswitch\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.387026 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385560 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-system-cni-dir\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.387026 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385584 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-host-run-netns\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.387026 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385607 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8snd\" (UniqueName: \"kubernetes.io/projected/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-kube-api-access-k8snd\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.387026 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385646 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-host-run-netns\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.387026 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385654 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-system-cni-dir\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.387026 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385672 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/18d4542b-53c1-473f-9605-c82fd5a642f4-ovn-node-metrics-cert\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.387026 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385696 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-kubernetes\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.387026 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385831 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/08f1ac32-f49e-4aa2-9ab8-8a1aac7bef36-konnectivity-ca\") pod \"konnectivity-agent-vpxpj\" (UID: \"08f1ac32-f49e-4aa2-9ab8-8a1aac7bef36\") " pod="kube-system/konnectivity-agent-vpxpj" Oct 30 21:13:06.387026 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385865 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-multus-conf-dir\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.387026 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385876 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-kubernetes\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.387859 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385888 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-run-netns\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.387859 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385912 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-var-lib-openvswitch\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.387859 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385934 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-log-socket\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.387859 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385940 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-multus-conf-dir\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.387859 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385956 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-run-ovn-kubernetes\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.387859 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385981 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-sysconfig\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.387859 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.385997 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-var-lib-openvswitch\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.387859 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386008 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25ca3b5e-7eba-4411-be09-5da0fe41e473-system-cni-dir\") pod \"multus-additional-cni-plugins-brt6h\" (UID: \"25ca3b5e-7eba-4411-be09-5da0fe41e473\") " pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:06.387859 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386017 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-run-netns\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.387859 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386042 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-run-ovn-kubernetes\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.387859 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386050 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25ca3b5e-7eba-4411-be09-5da0fe41e473-system-cni-dir\") pod \"multus-additional-cni-plugins-brt6h\" (UID: \"25ca3b5e-7eba-4411-be09-5da0fe41e473\") " pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:06.387859 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386066 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-log-socket\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.387859 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386068 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-host-var-lib-cni-bin\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.387859 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386099 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-host-run-multus-certs\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.387859 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386099 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-sysconfig\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.387859 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386153 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsrnm\" (UniqueName: \"kubernetes.io/projected/19538cda-3ba8-4bad-bb47-23a2511736e7-kube-api-access-bsrnm\") pod \"network-check-target-lw64d\" (UID: \"19538cda-3ba8-4bad-bb47-23a2511736e7\") " pod="openshift-network-diagnostics/network-check-target-lw64d" Oct 30 21:13:06.387859 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386166 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-host-var-lib-cni-bin\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.388666 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386178 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtvrj\" (UniqueName: \"kubernetes.io/projected/9597db44-1ca0-4fa8-ade7-cb55eeca9593-kube-api-access-wtvrj\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:06.388666 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386180 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-host-run-multus-certs\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.388666 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386200 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-lib-modules\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.388666 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386236 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-tuned\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.388666 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386279 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-cni-binary-copy\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.388666 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386277 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-lib-modules\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.388666 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386306 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54b8c0a5-e1a7-4d02-a402-b3cadc139e28-serviceca\") pod \"node-ca-q8g7b\" (UID: \"54b8c0a5-e1a7-4d02-a402-b3cadc139e28\") " pod="openshift-image-registry/node-ca-q8g7b" Oct 30 21:13:06.388666 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386358 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/08f1ac32-f49e-4aa2-9ab8-8a1aac7bef36-konnectivity-ca\") pod \"konnectivity-agent-vpxpj\" (UID: \"08f1ac32-f49e-4aa2-9ab8-8a1aac7bef36\") " pod="kube-system/konnectivity-agent-vpxpj" Oct 30 21:13:06.388666 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386369 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-kubelet\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.388666 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386397 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-sysctl-conf\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.388666 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386420 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/25ca3b5e-7eba-4411-be09-5da0fe41e473-cnibin\") pod \"multus-additional-cni-plugins-brt6h\" (UID: \"25ca3b5e-7eba-4411-be09-5da0fe41e473\") " pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:06.388666 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386445 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-etc-kubernetes\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.388666 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386472 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9597db44-1ca0-4fa8-ade7-cb55eeca9593-socket-dir\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:06.388666 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386500 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30bde564-b785-4359-9377-ffec880fb18c-metrics-certs\") pod \"network-metrics-daemon-n4g68\" (UID: \"30bde564-b785-4359-9377-ffec880fb18c\") " pod="openshift-multus/network-metrics-daemon-n4g68" Oct 30 21:13:06.388666 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386527 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/25ca3b5e-7eba-4411-be09-5da0fe41e473-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-brt6h\" (UID: \"25ca3b5e-7eba-4411-be09-5da0fe41e473\") " pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:06.388666 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386552 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-multus-cni-dir\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.388666 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386575 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3341bf47-8c8f-4073-8758-122e04e0b138-dbus\") pod \"global-pull-secret-syncer-jztdg\" (UID: \"3341bf47-8c8f-4073-8758-122e04e0b138\") " pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:06.389425 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386588 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/25ca3b5e-7eba-4411-be09-5da0fe41e473-cnibin\") pod \"multus-additional-cni-plugins-brt6h\" (UID: \"25ca3b5e-7eba-4411-be09-5da0fe41e473\") " pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:06.389425 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386598 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-cni-netd\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.389425 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386623 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cctxz\" (UniqueName: \"kubernetes.io/projected/05d96464-f946-4a83-88e5-ececfee30b4c-kube-api-access-cctxz\") pod \"node-resolver-p4c9m\" (UID: \"05d96464-f946-4a83-88e5-ececfee30b4c\") " pod="openshift-dns/node-resolver-p4c9m" Oct 30 21:13:06.389425 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386647 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-multus-daemon-config\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.389425 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386669 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54b8c0a5-e1a7-4d02-a402-b3cadc139e28-host\") pod \"node-ca-q8g7b\" (UID: \"54b8c0a5-e1a7-4d02-a402-b3cadc139e28\") " pod="openshift-image-registry/node-ca-q8g7b" Oct 30 21:13:06.389425 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386722 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/18d4542b-53c1-473f-9605-c82fd5a642f4-ovnkube-config\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.389425 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386763 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-sysctl-d\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.389425 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386788 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-sysctl-conf\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.389425 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386788 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05d96464-f946-4a83-88e5-ececfee30b4c-tmp-dir\") pod \"node-resolver-p4c9m\" (UID: \"05d96464-f946-4a83-88e5-ececfee30b4c\") " pod="openshift-dns/node-resolver-p4c9m" Oct 30 21:13:06.389425 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386832 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-host-run-k8s-cni-cncf-io\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.389425 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386830 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-cni-binary-copy\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.389425 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386859 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg66z\" (UniqueName: \"kubernetes.io/projected/54b8c0a5-e1a7-4d02-a402-b3cadc139e28-kube-api-access-jg66z\") pod \"node-ca-q8g7b\" (UID: \"54b8c0a5-e1a7-4d02-a402-b3cadc139e28\") " pod="openshift-image-registry/node-ca-q8g7b" Oct 30 21:13:06.389425 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386884 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-slash\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.389425 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386907 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-run-systemd\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.389425 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386951 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvgz2\" (UniqueName: \"kubernetes.io/projected/1d4aae53-afc9-4373-953d-b9b7c323e3ee-kube-api-access-tvgz2\") pod \"iptables-alerter-jfprx\" (UID: \"1d4aae53-afc9-4373-953d-b9b7c323e3ee\") " pod="openshift-network-operator/iptables-alerter-jfprx" Oct 30 21:13:06.389425 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386979 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/25ca3b5e-7eba-4411-be09-5da0fe41e473-cni-binary-copy\") pod \"multus-additional-cni-plugins-brt6h\" (UID: \"25ca3b5e-7eba-4411-be09-5da0fe41e473\") " pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:06.389425 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.387009 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/25ca3b5e-7eba-4411-be09-5da0fe41e473-tuning-conf-dir\") pod \"multus-additional-cni-plugins-brt6h\" (UID: \"25ca3b5e-7eba-4411-be09-5da0fe41e473\") " pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:06.390175 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.387029 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05d96464-f946-4a83-88e5-ececfee30b4c-tmp-dir\") pod \"node-resolver-p4c9m\" (UID: \"05d96464-f946-4a83-88e5-ececfee30b4c\") " pod="openshift-dns/node-resolver-p4c9m" Oct 30 21:13:06.390175 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.387034 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1d4aae53-afc9-4373-953d-b9b7c323e3ee-iptables-alerter-script\") pod \"iptables-alerter-jfprx\" (UID: \"1d4aae53-afc9-4373-953d-b9b7c323e3ee\") " pod="openshift-network-operator/iptables-alerter-jfprx" Oct 30 21:13:06.390175 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.387061 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl9th\" (UniqueName: \"kubernetes.io/projected/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-kube-api-access-hl9th\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.390175 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.387086 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-hostroot\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.390175 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.387107 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-run\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.390175 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.387113 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-etc-kubernetes\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.390175 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.387133 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-var-lib-kubelet\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.390175 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.387177 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sqpcg\"" Oct 30 21:13:06.390175 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.387214 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-var-lib-kubelet\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.390175 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.387214 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9597db44-1ca0-4fa8-ade7-cb55eeca9593-socket-dir\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:06.390175 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.387232 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:13:06.390175 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.387259 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-host-run-k8s-cni-cncf-io\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.390175 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:06.387284 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:13:06.390175 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:06.387333 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30bde564-b785-4359-9377-ffec880fb18c-metrics-certs podName:30bde564-b785-4359-9377-ffec880fb18c nodeName:}" failed. No retries permitted until 2026-10-30 21:13:06.887315373 +0000 UTC m=+2.324309905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30bde564-b785-4359-9377-ffec880fb18c-metrics-certs") pod "network-metrics-daemon-n4g68" (UID: "30bde564-b785-4359-9377-ffec880fb18c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:13:06.390175 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.387381 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-h52wm\"" Oct 30 21:13:06.390175 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.387472 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-slash\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.390175 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.387520 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-run-systemd\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.390175 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.387524 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-79sqh\"" Oct 30 21:13:06.390175 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.387700 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4v9xp\"" Oct 30 21:13:06.390175 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.387872 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/25ca3b5e-7eba-4411-be09-5da0fe41e473-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-brt6h\" (UID: \"25ca3b5e-7eba-4411-be09-5da0fe41e473\") " pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:06.391039 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.388008 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-multus-cni-dir\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.391039 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.388066 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3341bf47-8c8f-4073-8758-122e04e0b138-dbus\") pod \"global-pull-secret-syncer-jztdg\" (UID: \"3341bf47-8c8f-4073-8758-122e04e0b138\") " pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:06.391039 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.388121 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-cni-netd\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.391039 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.388214 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/25ca3b5e-7eba-4411-be09-5da0fe41e473-cni-binary-copy\") pod \"multus-additional-cni-plugins-brt6h\" (UID: \"25ca3b5e-7eba-4411-be09-5da0fe41e473\") " pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:06.391039 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.388299 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:13:06.391039 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.388381 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:13:06.391039 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.388403 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/25ca3b5e-7eba-4411-be09-5da0fe41e473-tuning-conf-dir\") pod \"multus-additional-cni-plugins-brt6h\" (UID: \"25ca3b5e-7eba-4411-be09-5da0fe41e473\") " pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:06.391039 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.388436 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:13:06.391039 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.388639 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:13:06.391039 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.388712 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-hostroot\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.391039 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.388773 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-run\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.391039 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.388804 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:13:06.391039 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.386669 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-kubelet\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.391039 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.388957 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54b8c0a5-e1a7-4d02-a402-b3cadc139e28-host\") pod \"node-ca-q8g7b\" (UID: \"54b8c0a5-e1a7-4d02-a402-b3cadc139e28\") " pod="openshift-image-registry/node-ca-q8g7b" Oct 30 21:13:06.391039 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.388969 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-tuned\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.391039 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.388991 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/08f1ac32-f49e-4aa2-9ab8-8a1aac7bef36-agent-certs\") pod \"konnectivity-agent-vpxpj\" (UID: \"08f1ac32-f49e-4aa2-9ab8-8a1aac7bef36\") " pod="kube-system/konnectivity-agent-vpxpj" Oct 30 21:13:06.391039 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.389003 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:13:06.391039 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.389056 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-sysctl-d\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.391039 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.389094 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:13:06.391039 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.389137 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:13:06.391039 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.389158 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:13:06.391039 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.389193 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:13:06.391039 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.389325 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:13:06.391039 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.389548 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/18d4542b-53c1-473f-9605-c82fd5a642f4-ovnkube-config\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.391806 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.389704 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:13:06.391806 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.389696 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:13:06.391806 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.389703 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-tmp\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.395856 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.395839 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/18d4542b-53c1-473f-9605-c82fd5a642f4-env-overrides\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.395910 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.395893 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/18d4542b-53c1-473f-9605-c82fd5a642f4-ovnkube-script-lib\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.397413 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.397394 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54b8c0a5-e1a7-4d02-a402-b3cadc139e28-serviceca\") pod \"node-ca-q8g7b\" (UID: \"54b8c0a5-e1a7-4d02-a402-b3cadc139e28\") " pod="openshift-image-registry/node-ca-q8g7b" Oct 30 21:13:06.397586 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.397570 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/18d4542b-53c1-473f-9605-c82fd5a642f4-ovn-node-metrics-cert\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.399756 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.399734 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1d4aae53-afc9-4373-953d-b9b7c323e3ee-iptables-alerter-script\") pod \"iptables-alerter-jfprx\" (UID: \"1d4aae53-afc9-4373-953d-b9b7c323e3ee\") " pod="openshift-network-operator/iptables-alerter-jfprx" Oct 30 21:13:06.399990 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.399971 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-multus-daemon-config\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.447695 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.447637 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8884\" (UniqueName: \"kubernetes.io/projected/30bde564-b785-4359-9377-ffec880fb18c-kube-api-access-l8884\") pod \"network-metrics-daemon-n4g68\" (UID: \"30bde564-b785-4359-9377-ffec880fb18c\") " pod="openshift-multus/network-metrics-daemon-n4g68" Oct 30 21:13:06.449828 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.449800 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvbqm\" (UniqueName: \"kubernetes.io/projected/25ca3b5e-7eba-4411-be09-5da0fe41e473-kube-api-access-mvbqm\") pod \"multus-additional-cni-plugins-brt6h\" (UID: \"25ca3b5e-7eba-4411-be09-5da0fe41e473\") " pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:06.449905 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.449801 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtvrj\" (UniqueName: \"kubernetes.io/projected/9597db44-1ca0-4fa8-ade7-cb55eeca9593-kube-api-access-wtvrj\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:06.450406 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:06.450390 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:13:06.450491 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:06.450409 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:13:06.450491 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:06.450422 2577 projected.go:196] Error preparing data for projected volume kube-api-access-bsrnm for pod openshift-network-diagnostics/network-check-target-lw64d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:13:06.450599 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:06.450491 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19538cda-3ba8-4bad-bb47-23a2511736e7-kube-api-access-bsrnm podName:19538cda-3ba8-4bad-bb47-23a2511736e7 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:06.950473572 +0000 UTC m=+2.387468107 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bsrnm" (UniqueName: "kubernetes.io/projected/19538cda-3ba8-4bad-bb47-23a2511736e7-kube-api-access-bsrnm") pod "network-check-target-lw64d" (UID: "19538cda-3ba8-4bad-bb47-23a2511736e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:13:06.450781 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.450765 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg66z\" (UniqueName: \"kubernetes.io/projected/54b8c0a5-e1a7-4d02-a402-b3cadc139e28-kube-api-access-jg66z\") pod \"node-ca-q8g7b\" (UID: \"54b8c0a5-e1a7-4d02-a402-b3cadc139e28\") " pod="openshift-image-registry/node-ca-q8g7b" Oct 30 21:13:06.451105 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.451090 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8snd\" (UniqueName: \"kubernetes.io/projected/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-kube-api-access-k8snd\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.454953 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.454933 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cctxz\" (UniqueName: \"kubernetes.io/projected/05d96464-f946-4a83-88e5-ececfee30b4c-kube-api-access-cctxz\") pod \"node-resolver-p4c9m\" (UID: \"05d96464-f946-4a83-88e5-ececfee30b4c\") " pod="openshift-dns/node-resolver-p4c9m" Oct 30 21:13:06.455045 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.455009 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k52xf\" (UniqueName: \"kubernetes.io/projected/18d4542b-53c1-473f-9605-c82fd5a642f4-kube-api-access-k52xf\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.455080 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.455058 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl9th\" (UniqueName: \"kubernetes.io/projected/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-kube-api-access-hl9th\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.455881 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.455867 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvgz2\" (UniqueName: \"kubernetes.io/projected/1d4aae53-afc9-4373-953d-b9b7c323e3ee-kube-api-access-tvgz2\") pod \"iptables-alerter-jfprx\" (UID: \"1d4aae53-afc9-4373-953d-b9b7c323e3ee\") " pod="openshift-network-operator/iptables-alerter-jfprx" Oct 30 21:13:06.559180 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.559152 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:06.564425 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.564407 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:13:06.572655 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.572636 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:06.613373 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.613335 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:06.616975 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.616957 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vpxpj" Oct 30 21:13:06.624470 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.624452 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p4c9m" Oct 30 21:13:06.657167 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.657138 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:06.659266 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.659247 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zn2wp" Oct 30 21:13:06.675870 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.675846 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jfprx" Oct 30 21:13:06.680312 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.680299 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q8g7b" Oct 30 21:13:06.710331 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:06.710294 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b9addd42169977c49b17223d2454db0.slice/crio-81378ec2ad2c76d0f97979360af798bf2d824bde7db7842663a8373b224f5bb0 WatchSource:0}: Error finding container 81378ec2ad2c76d0f97979360af798bf2d824bde7db7842663a8373b224f5bb0: Status 404 returned error can't find the container with id 81378ec2ad2c76d0f97979360af798bf2d824bde7db7842663a8373b224f5bb0 Oct 30 21:13:06.716198 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.716186 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:13:06.771658 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:06.771635 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e151e7c539d69d81e686cdbfb721e69.slice/crio-0bc90de4c74ac62133ee2f84e8d2364207b45a0853fc7b80529262f8213ce9c2 WatchSource:0}: Error finding container 0bc90de4c74ac62133ee2f84e8d2364207b45a0853fc7b80529262f8213ce9c2: Status 404 returned error can't find the container with id 0bc90de4c74ac62133ee2f84e8d2364207b45a0853fc7b80529262f8213ce9c2 Oct 30 21:13:06.890137 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.890106 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3341bf47-8c8f-4073-8758-122e04e0b138-original-pull-secret\") pod \"global-pull-secret-syncer-jztdg\" (UID: \"3341bf47-8c8f-4073-8758-122e04e0b138\") " pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:06.890321 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.890170 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30bde564-b785-4359-9377-ffec880fb18c-metrics-certs\") pod \"network-metrics-daemon-n4g68\" (UID: \"30bde564-b785-4359-9377-ffec880fb18c\") " pod="openshift-multus/network-metrics-daemon-n4g68" Oct 30 21:13:06.890321 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:06.890240 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:13:06.890321 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:06.890271 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:13:06.890321 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:06.890293 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3341bf47-8c8f-4073-8758-122e04e0b138-original-pull-secret podName:3341bf47-8c8f-4073-8758-122e04e0b138 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:07.890279587 +0000 UTC m=+3.327274117 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3341bf47-8c8f-4073-8758-122e04e0b138-original-pull-secret") pod "global-pull-secret-syncer-jztdg" (UID: "3341bf47-8c8f-4073-8758-122e04e0b138") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:13:06.890321 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:06.890314 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30bde564-b785-4359-9377-ffec880fb18c-metrics-certs podName:30bde564-b785-4359-9377-ffec880fb18c nodeName:}" failed. No retries permitted until 2026-10-30 21:13:07.890301011 +0000 UTC m=+3.327295543 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30bde564-b785-4359-9377-ffec880fb18c-metrics-certs") pod "network-metrics-daemon-n4g68" (UID: "30bde564-b785-4359-9377-ffec880fb18c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:13:06.891847 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.891832 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:13:06.991455 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:06.991388 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsrnm\" (UniqueName: \"kubernetes.io/projected/19538cda-3ba8-4bad-bb47-23a2511736e7-kube-api-access-bsrnm\") pod \"network-check-target-lw64d\" (UID: \"19538cda-3ba8-4bad-bb47-23a2511736e7\") " pod="openshift-network-diagnostics/network-check-target-lw64d" Oct 30 21:13:06.991930 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:06.991500 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:13:06.991930 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:06.991517 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:13:06.991930 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:06.991526 2577 projected.go:196] Error preparing data for projected volume kube-api-access-bsrnm for pod openshift-network-diagnostics/network-check-target-lw64d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:13:06.991930 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:06.991568 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19538cda-3ba8-4bad-bb47-23a2511736e7-kube-api-access-bsrnm podName:19538cda-3ba8-4bad-bb47-23a2511736e7 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:07.991557739 +0000 UTC m=+3.428552257 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bsrnm" (UniqueName: "kubernetes.io/projected/19538cda-3ba8-4bad-bb47-23a2511736e7-kube-api-access-bsrnm") pod "network-check-target-lw64d" (UID: "19538cda-3ba8-4bad-bb47-23a2511736e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:13:07.120403 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:07.120350 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal" event={"ID":"2e151e7c539d69d81e686cdbfb721e69","Type":"ContainerStarted","Data":"0bc90de4c74ac62133ee2f84e8d2364207b45a0853fc7b80529262f8213ce9c2"} Oct 30 21:13:07.121283 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:07.121260 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-83.us-west-2.compute.internal" event={"ID":"8b9addd42169977c49b17223d2454db0","Type":"ContainerStarted","Data":"81378ec2ad2c76d0f97979360af798bf2d824bde7db7842663a8373b224f5bb0"} Oct 30 21:13:07.371353 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:07.371263 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9597db44_1ca0_4fa8_ade7_cb55eeca9593.slice/crio-9a77293a9478bf036b3af7ffc7cb8489a1d116738bb93f46336e37ed1c9b7506 WatchSource:0}: Error finding container 9a77293a9478bf036b3af7ffc7cb8489a1d116738bb93f46336e37ed1c9b7506: Status 404 returned error can't find the container with id 9a77293a9478bf036b3af7ffc7cb8489a1d116738bb93f46336e37ed1c9b7506 Oct 30 21:13:07.384625 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:07.384578 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:08:06 +0000 UTC" deadline="2028-04-27 08:27:49.607455642 +0000 UTC" Oct 30 21:13:07.384625 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:07.384618 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13067h14m42.222840512s" Oct 30 21:13:07.400217 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:07.400188 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08f1ac32_f49e_4aa2_9ab8_8a1aac7bef36.slice/crio-c580ec59787cb3cc0e10e37be16b38aa039208e54b726780cea4cf2f5d570fb2 WatchSource:0}: Error finding container c580ec59787cb3cc0e10e37be16b38aa039208e54b726780cea4cf2f5d570fb2: Status 404 returned error can't find the container with id c580ec59787cb3cc0e10e37be16b38aa039208e54b726780cea4cf2f5d570fb2 Oct 30 21:13:07.449112 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:07.449077 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25ca3b5e_7eba_4411_be09_5da0fe41e473.slice/crio-15d280b6927840bc8d5e670ceaf0d1ad63c76214be7bdfb29b41789dfb1e849b WatchSource:0}: Error finding container 15d280b6927840bc8d5e670ceaf0d1ad63c76214be7bdfb29b41789dfb1e849b: Status 404 returned error can't find the container with id 15d280b6927840bc8d5e670ceaf0d1ad63c76214be7bdfb29b41789dfb1e849b Oct 30 21:13:07.449983 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:07.449959 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e3a1e7_fed8_4969_b9ae_baec7c2dade8.slice/crio-0d97eb97d73cf0500bdd6be5473271d95ca4db1cd263615fd197bf7b6a7c12d0 WatchSource:0}: Error finding container 0d97eb97d73cf0500bdd6be5473271d95ca4db1cd263615fd197bf7b6a7c12d0: Status 404 returned error can't find the container with id 0d97eb97d73cf0500bdd6be5473271d95ca4db1cd263615fd197bf7b6a7c12d0 Oct 30 21:13:07.488845 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:07.488815 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c877970_90d9_43ba_9c9b_4a22fbda1dc0.slice/crio-11f55cd81c2b0cd75bad91a53655e3dcbf2481f2269d44de8a43f7d6489a5d66 WatchSource:0}: Error finding container 11f55cd81c2b0cd75bad91a53655e3dcbf2481f2269d44de8a43f7d6489a5d66: Status 404 returned error can't find the container with id 11f55cd81c2b0cd75bad91a53655e3dcbf2481f2269d44de8a43f7d6489a5d66 Oct 30 21:13:07.525997 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:07.525970 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d4aae53_afc9_4373_953d_b9b7c323e3ee.slice/crio-dd42db4be3a7cb80a93a5bf0a381856b58cae9e07d84532c87a1eab458a25b76 WatchSource:0}: Error finding container dd42db4be3a7cb80a93a5bf0a381856b58cae9e07d84532c87a1eab458a25b76: Status 404 returned error can't find the container with id dd42db4be3a7cb80a93a5bf0a381856b58cae9e07d84532c87a1eab458a25b76 Oct 30 21:13:07.528201 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:07.528113 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05d96464_f946_4a83_88e5_ececfee30b4c.slice/crio-390843099a133d9ae571a91d7988a0eb49448d83668b3a037d62a0de90d0396f WatchSource:0}: Error finding container 390843099a133d9ae571a91d7988a0eb49448d83668b3a037d62a0de90d0396f: Status 404 returned error can't find the container with id 390843099a133d9ae571a91d7988a0eb49448d83668b3a037d62a0de90d0396f Oct 30 21:13:07.530800 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:07.530777 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54b8c0a5_e1a7_4d02_a402_b3cadc139e28.slice/crio-d1d8eebf965a999029a65e81b2c59c8511529ef74c8e1ff1c67290be1b4fdb05 WatchSource:0}: Error finding container d1d8eebf965a999029a65e81b2c59c8511529ef74c8e1ff1c67290be1b4fdb05: Status 404 returned error can't find the container with id d1d8eebf965a999029a65e81b2c59c8511529ef74c8e1ff1c67290be1b4fdb05 Oct 30 21:13:07.601930 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:07.601907 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18d4542b_53c1_473f_9605_c82fd5a642f4.slice/crio-caacc81147f4b137a7f1f62412f4b237d9c264e996823f9897bf2da9577056c3 WatchSource:0}: Error finding container caacc81147f4b137a7f1f62412f4b237d9c264e996823f9897bf2da9577056c3: Status 404 returned error can't find the container with id caacc81147f4b137a7f1f62412f4b237d9c264e996823f9897bf2da9577056c3 Oct 30 21:13:07.897587 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:07.897414 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3341bf47-8c8f-4073-8758-122e04e0b138-original-pull-secret\") pod \"global-pull-secret-syncer-jztdg\" (UID: \"3341bf47-8c8f-4073-8758-122e04e0b138\") " pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:07.897587 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:07.897489 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30bde564-b785-4359-9377-ffec880fb18c-metrics-certs\") pod \"network-metrics-daemon-n4g68\" (UID: \"30bde564-b785-4359-9377-ffec880fb18c\") " pod="openshift-multus/network-metrics-daemon-n4g68" Oct 30 21:13:07.897587 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:07.897570 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:13:07.897901 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:07.897627 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3341bf47-8c8f-4073-8758-122e04e0b138-original-pull-secret podName:3341bf47-8c8f-4073-8758-122e04e0b138 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:09.897610825 +0000 UTC m=+5.334605348 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3341bf47-8c8f-4073-8758-122e04e0b138-original-pull-secret") pod "global-pull-secret-syncer-jztdg" (UID: "3341bf47-8c8f-4073-8758-122e04e0b138") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:13:07.897901 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:07.897570 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:13:07.897901 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:07.897665 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30bde564-b785-4359-9377-ffec880fb18c-metrics-certs podName:30bde564-b785-4359-9377-ffec880fb18c nodeName:}" failed. No retries permitted until 2026-10-30 21:13:09.897655663 +0000 UTC m=+5.334650181 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30bde564-b785-4359-9377-ffec880fb18c-metrics-certs") pod "network-metrics-daemon-n4g68" (UID: "30bde564-b785-4359-9377-ffec880fb18c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:13:07.998442 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:07.998405 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsrnm\" (UniqueName: \"kubernetes.io/projected/19538cda-3ba8-4bad-bb47-23a2511736e7-kube-api-access-bsrnm\") pod \"network-check-target-lw64d\" (UID: \"19538cda-3ba8-4bad-bb47-23a2511736e7\") " pod="openshift-network-diagnostics/network-check-target-lw64d" Oct 30 21:13:07.999100 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:07.998616 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:13:07.999100 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:07.998640 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:13:07.999100 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:07.998653 2577 projected.go:196] Error preparing data for projected volume kube-api-access-bsrnm for pod openshift-network-diagnostics/network-check-target-lw64d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:13:07.999100 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:07.998708 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19538cda-3ba8-4bad-bb47-23a2511736e7-kube-api-access-bsrnm podName:19538cda-3ba8-4bad-bb47-23a2511736e7 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:09.998691087 +0000 UTC m=+5.435685609 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bsrnm" (UniqueName: "kubernetes.io/projected/19538cda-3ba8-4bad-bb47-23a2511736e7-kube-api-access-bsrnm") pod "network-check-target-lw64d" (UID: "19538cda-3ba8-4bad-bb47-23a2511736e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:13:08.117013 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:08.116979 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lw64d" Oct 30 21:13:08.117174 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:08.116993 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:08.117174 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:08.117098 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4g68" Oct 30 21:13:08.117174 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:08.117103 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lw64d" podUID="19538cda-3ba8-4bad-bb47-23a2511736e7" Oct 30 21:13:08.117174 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:08.117164 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jztdg" podUID="3341bf47-8c8f-4073-8758-122e04e0b138" Oct 30 21:13:08.117374 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:08.117233 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4g68" podUID="30bde564-b785-4359-9377-ffec880fb18c" Oct 30 21:13:08.124840 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:08.124798 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" event={"ID":"18d4542b-53c1-473f-9605-c82fd5a642f4","Type":"ContainerStarted","Data":"caacc81147f4b137a7f1f62412f4b237d9c264e996823f9897bf2da9577056c3"} Oct 30 21:13:08.126517 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:08.126484 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q8g7b" event={"ID":"54b8c0a5-e1a7-4d02-a402-b3cadc139e28","Type":"ContainerStarted","Data":"d1d8eebf965a999029a65e81b2c59c8511529ef74c8e1ff1c67290be1b4fdb05"} Oct 30 21:13:08.127843 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:08.127817 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p4c9m" event={"ID":"05d96464-f946-4a83-88e5-ececfee30b4c","Type":"ContainerStarted","Data":"390843099a133d9ae571a91d7988a0eb49448d83668b3a037d62a0de90d0396f"} Oct 30 21:13:08.129221 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:08.129196 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" event={"ID":"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8","Type":"ContainerStarted","Data":"0d97eb97d73cf0500bdd6be5473271d95ca4db1cd263615fd197bf7b6a7c12d0"} Oct 30 21:13:08.130804 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:08.130780 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-brt6h" event={"ID":"25ca3b5e-7eba-4411-be09-5da0fe41e473","Type":"ContainerStarted","Data":"15d280b6927840bc8d5e670ceaf0d1ad63c76214be7bdfb29b41789dfb1e849b"} Oct 30 21:13:08.132102 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:08.132072 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vpxpj" event={"ID":"08f1ac32-f49e-4aa2-9ab8-8a1aac7bef36","Type":"ContainerStarted","Data":"c580ec59787cb3cc0e10e37be16b38aa039208e54b726780cea4cf2f5d570fb2"} Oct 30 21:13:08.133302 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:08.133281 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jfprx" event={"ID":"1d4aae53-afc9-4373-953d-b9b7c323e3ee","Type":"ContainerStarted","Data":"dd42db4be3a7cb80a93a5bf0a381856b58cae9e07d84532c87a1eab458a25b76"} Oct 30 21:13:08.134817 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:08.134794 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zn2wp" event={"ID":"2c877970-90d9-43ba-9c9b-4a22fbda1dc0","Type":"ContainerStarted","Data":"11f55cd81c2b0cd75bad91a53655e3dcbf2481f2269d44de8a43f7d6489a5d66"} Oct 30 21:13:08.136417 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:08.136397 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" event={"ID":"9597db44-1ca0-4fa8-ade7-cb55eeca9593","Type":"ContainerStarted","Data":"9a77293a9478bf036b3af7ffc7cb8489a1d116738bb93f46336e37ed1c9b7506"} Oct 30 21:13:08.385477 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:08.385287 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:08:06 +0000 UTC" deadline="2028-08-12 04:02:06.2116376 +0000 UTC" Oct 30 21:13:08.385477 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:08.385367 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15630h48m57.826275553s" Oct 30 21:13:09.915402 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:09.915361 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3341bf47-8c8f-4073-8758-122e04e0b138-original-pull-secret\") pod \"global-pull-secret-syncer-jztdg\" (UID: \"3341bf47-8c8f-4073-8758-122e04e0b138\") " pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:09.915916 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:09.915449 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30bde564-b785-4359-9377-ffec880fb18c-metrics-certs\") pod \"network-metrics-daemon-n4g68\" (UID: \"30bde564-b785-4359-9377-ffec880fb18c\") " pod="openshift-multus/network-metrics-daemon-n4g68" Oct 30 21:13:09.915916 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:09.915597 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:13:09.915916 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:09.915668 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30bde564-b785-4359-9377-ffec880fb18c-metrics-certs podName:30bde564-b785-4359-9377-ffec880fb18c nodeName:}" failed. No retries permitted until 2026-10-30 21:13:13.915650549 +0000 UTC m=+9.352645089 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30bde564-b785-4359-9377-ffec880fb18c-metrics-certs") pod "network-metrics-daemon-n4g68" (UID: "30bde564-b785-4359-9377-ffec880fb18c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:13:09.915916 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:09.915746 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:13:09.915916 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:09.915783 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3341bf47-8c8f-4073-8758-122e04e0b138-original-pull-secret podName:3341bf47-8c8f-4073-8758-122e04e0b138 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:13.915771349 +0000 UTC m=+9.352765870 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3341bf47-8c8f-4073-8758-122e04e0b138-original-pull-secret") pod "global-pull-secret-syncer-jztdg" (UID: "3341bf47-8c8f-4073-8758-122e04e0b138") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:13:10.016519 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:10.016478 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsrnm\" (UniqueName: \"kubernetes.io/projected/19538cda-3ba8-4bad-bb47-23a2511736e7-kube-api-access-bsrnm\") pod \"network-check-target-lw64d\" (UID: \"19538cda-3ba8-4bad-bb47-23a2511736e7\") " pod="openshift-network-diagnostics/network-check-target-lw64d" Oct 30 21:13:10.016705 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:10.016672 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:13:10.016705 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:10.016692 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:13:10.016705 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:10.016706 2577 projected.go:196] Error preparing data for projected volume kube-api-access-bsrnm for pod openshift-network-diagnostics/network-check-target-lw64d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:13:10.016873 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:10.016761 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19538cda-3ba8-4bad-bb47-23a2511736e7-kube-api-access-bsrnm podName:19538cda-3ba8-4bad-bb47-23a2511736e7 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:14.016744526 +0000 UTC m=+9.453739047 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bsrnm" (UniqueName: "kubernetes.io/projected/19538cda-3ba8-4bad-bb47-23a2511736e7-kube-api-access-bsrnm") pod "network-check-target-lw64d" (UID: "19538cda-3ba8-4bad-bb47-23a2511736e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:13:10.118959 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:10.118923 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4g68" Oct 30 21:13:10.119106 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:10.119060 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4g68" podUID="30bde564-b785-4359-9377-ffec880fb18c" Oct 30 21:13:10.119148 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:10.119114 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lw64d" Oct 30 21:13:10.119201 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:10.119176 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lw64d" podUID="19538cda-3ba8-4bad-bb47-23a2511736e7" Oct 30 21:13:10.119245 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:10.119226 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:10.119303 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:10.119288 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jztdg" podUID="3341bf47-8c8f-4073-8758-122e04e0b138" Oct 30 21:13:11.166255 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:11.166049 2577 generic.go:356] "Generic (PLEG): container finished" podID="25ca3b5e-7eba-4411-be09-5da0fe41e473" containerID="78406c1ea5663373189fa90efef3f9fe30b11084885bc25ee2e459f7aebb2718" exitCode=0 Oct 30 21:13:11.166726 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:11.166121 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-brt6h" event={"ID":"25ca3b5e-7eba-4411-be09-5da0fe41e473","Type":"ContainerDied","Data":"78406c1ea5663373189fa90efef3f9fe30b11084885bc25ee2e459f7aebb2718"} Oct 30 21:13:11.179433 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:11.179380 2577 generic.go:356] "Generic (PLEG): container finished" podID="2e151e7c539d69d81e686cdbfb721e69" containerID="20aed17bbb7fa67e394737034e85203c006f8e6d156a208c62d502d17a1bbe4b" exitCode=0 Oct 30 21:13:11.179933 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:11.179915 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal" event={"ID":"2e151e7c539d69d81e686cdbfb721e69","Type":"ContainerDied","Data":"20aed17bbb7fa67e394737034e85203c006f8e6d156a208c62d502d17a1bbe4b"} Oct 30 21:13:11.201559 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:11.201534 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-83.us-west-2.compute.internal" event={"ID":"8b9addd42169977c49b17223d2454db0","Type":"ContainerStarted","Data":"3d1dd588a97dc824f48fd65376ffd16339e7276d35ce8687df74047bd3fc2a91"} Oct 30 21:13:11.527098 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:11.527038 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-83.us-west-2.compute.internal" podStartSLOduration=6.527018367 podStartE2EDuration="6.527018367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:13:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:11.407733405 +0000 UTC m=+6.844727946" watchObservedRunningTime="2026-10-30 21:13:11.527018367 +0000 UTC m=+6.964012910" Oct 30 21:13:12.117833 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:12.117173 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:12.117833 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:12.117189 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lw64d" Oct 30 21:13:12.117833 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:12.117297 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jztdg" podUID="3341bf47-8c8f-4073-8758-122e04e0b138" Oct 30 21:13:12.117833 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:12.117633 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4g68" Oct 30 21:13:12.117833 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:12.117724 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lw64d" podUID="19538cda-3ba8-4bad-bb47-23a2511736e7" Oct 30 21:13:12.117833 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:12.117805 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4g68" podUID="30bde564-b785-4359-9377-ffec880fb18c" Oct 30 21:13:12.208484 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:12.208447 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vpxpj" event={"ID":"08f1ac32-f49e-4aa2-9ab8-8a1aac7bef36","Type":"ContainerStarted","Data":"c86f9bdb960ba40951adf73c480c02cdc9b06b0fcbf86f6063a5da0ad79e91d9"} Oct 30 21:13:12.216115 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:12.215992 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal" event={"ID":"2e151e7c539d69d81e686cdbfb721e69","Type":"ContainerStarted","Data":"6bb4891ae70f10c63db27dd956f9377bebab1778b0eef09d1d00221b23763a08"} Oct 30 21:13:12.442691 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:12.442639 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal" podStartSLOduration=7.442619361 podStartE2EDuration="7.442619361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:13:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:12.442433082 +0000 UTC m=+7.879427621" watchObservedRunningTime="2026-10-30 21:13:12.442619361 +0000 UTC m=+7.879613901" Oct 30 21:13:12.443004 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:12.442978 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vpxpj" podStartSLOduration=2.877281623 podStartE2EDuration="7.442971166s" totalImagesPullingTime="4.565689543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:12.325760853 +0000 UTC m=+7.762755394" watchObservedRunningTime="2026-10-30 21:13:12.442971166 +0000 UTC m=+7.879965706" Oct 30 21:13:13.953557 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:13.953475 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3341bf47-8c8f-4073-8758-122e04e0b138-original-pull-secret\") pod \"global-pull-secret-syncer-jztdg\" (UID: \"3341bf47-8c8f-4073-8758-122e04e0b138\") " pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:13.953557 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:13.953550 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30bde564-b785-4359-9377-ffec880fb18c-metrics-certs\") pod \"network-metrics-daemon-n4g68\" (UID: \"30bde564-b785-4359-9377-ffec880fb18c\") " pod="openshift-multus/network-metrics-daemon-n4g68" Oct 30 21:13:13.954062 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:13.953652 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:13:13.954062 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:13.953672 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:13:13.954062 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:13.953726 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3341bf47-8c8f-4073-8758-122e04e0b138-original-pull-secret podName:3341bf47-8c8f-4073-8758-122e04e0b138 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:21.953704233 +0000 UTC m=+17.390698762 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3341bf47-8c8f-4073-8758-122e04e0b138-original-pull-secret") pod "global-pull-secret-syncer-jztdg" (UID: "3341bf47-8c8f-4073-8758-122e04e0b138") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:13:13.954062 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:13.953746 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30bde564-b785-4359-9377-ffec880fb18c-metrics-certs podName:30bde564-b785-4359-9377-ffec880fb18c nodeName:}" failed. No retries permitted until 2026-10-30 21:13:21.953737435 +0000 UTC m=+17.390731953 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30bde564-b785-4359-9377-ffec880fb18c-metrics-certs") pod "network-metrics-daemon-n4g68" (UID: "30bde564-b785-4359-9377-ffec880fb18c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:13:14.054477 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:14.054175 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsrnm\" (UniqueName: \"kubernetes.io/projected/19538cda-3ba8-4bad-bb47-23a2511736e7-kube-api-access-bsrnm\") pod \"network-check-target-lw64d\" (UID: \"19538cda-3ba8-4bad-bb47-23a2511736e7\") " pod="openshift-network-diagnostics/network-check-target-lw64d" Oct 30 21:13:14.054477 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:14.054347 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:13:14.054477 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:14.054371 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:13:14.054477 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:14.054384 2577 projected.go:196] Error preparing data for projected volume kube-api-access-bsrnm for pod openshift-network-diagnostics/network-check-target-lw64d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:13:14.054477 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:14.054438 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19538cda-3ba8-4bad-bb47-23a2511736e7-kube-api-access-bsrnm podName:19538cda-3ba8-4bad-bb47-23a2511736e7 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:22.054421106 +0000 UTC m=+17.491415639 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bsrnm" (UniqueName: "kubernetes.io/projected/19538cda-3ba8-4bad-bb47-23a2511736e7-kube-api-access-bsrnm") pod "network-check-target-lw64d" (UID: "19538cda-3ba8-4bad-bb47-23a2511736e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:13:14.118028 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:14.117640 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4g68" Oct 30 21:13:14.118028 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:14.117785 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4g68" podUID="30bde564-b785-4359-9377-ffec880fb18c" Oct 30 21:13:14.118028 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:14.117851 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lw64d" Oct 30 21:13:14.118028 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:14.117896 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lw64d" podUID="19538cda-3ba8-4bad-bb47-23a2511736e7" Oct 30 21:13:14.118028 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:14.117936 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:14.118028 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:14.117977 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jztdg" podUID="3341bf47-8c8f-4073-8758-122e04e0b138" Oct 30 21:13:14.225724 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:14.225596 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" event={"ID":"9597db44-1ca0-4fa8-ade7-cb55eeca9593","Type":"ContainerStarted","Data":"dbe04007d0b7d36ffa67d6949fdceeafe33168f748b858430bb9a8a88998f392"} Oct 30 21:13:14.228661 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:14.227958 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q8g7b" event={"ID":"54b8c0a5-e1a7-4d02-a402-b3cadc139e28","Type":"ContainerStarted","Data":"c18b23f7860b9408a42a0b7243b55ff6d99198d9fb66840d7e8d10ab42996335"} Oct 30 21:13:16.054114 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:16.054090 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vpxpj" Oct 30 21:13:16.055216 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:16.055034 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vpxpj" Oct 30 21:13:16.063412 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:16.063257 2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:13:16.117423 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:16.117393 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4g68" Oct 30 21:13:16.117592 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:16.117532 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4g68" podUID="30bde564-b785-4359-9377-ffec880fb18c" Oct 30 21:13:16.117669 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:16.117607 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lw64d" Oct 30 21:13:16.117725 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:16.117678 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lw64d" podUID="19538cda-3ba8-4bad-bb47-23a2511736e7" Oct 30 21:13:16.117776 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:16.117733 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:16.117824 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:16.117801 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jztdg" podUID="3341bf47-8c8f-4073-8758-122e04e0b138" Oct 30 21:13:16.172412 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:16.172009 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-q8g7b" podStartSLOduration=5.526065877 podStartE2EDuration="11.171992641s" totalImagesPullingTime="5.645926764s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:13:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:14.345656837 +0000 UTC m=+9.782651378" watchObservedRunningTime="2026-10-30 21:13:16.171992641 +0000 UTC m=+11.608987182" Oct 30 21:13:16.242600 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:16.242384 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zn2wp" event={"ID":"2c877970-90d9-43ba-9c9b-4a22fbda1dc0","Type":"ContainerStarted","Data":"ad382fc30127c6653e51e14e574d89bbbc90e27bf9c0a438409493ba9b82ca39"} Oct 30 21:13:16.245021 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:16.244481 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" event={"ID":"9597db44-1ca0-4fa8-ade7-cb55eeca9593","Type":"ContainerStarted","Data":"2639048ba38bfe3b0dc0b478c0db13bf3053be5e50e742dd4353a0a35d1d7cc5"} Oct 30 21:13:16.245352 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:16.245289 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vpxpj" Oct 30 21:13:16.245761 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:16.245728 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vpxpj" Oct 30 21:13:16.479526 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:16.479259 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-zn2wp" podStartSLOduration=3.638830523 podStartE2EDuration="11.479239808s" totalImagesPullingTime="7.840409285s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:13:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:16.363037717 +0000 UTC m=+11.800032260" watchObservedRunningTime="2026-10-30 21:13:16.479239808 +0000 UTC m=+11.916234348" Oct 30 21:13:17.058945 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:17.058821 2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:13:16.063275437Z","UUID":"baea279b-6128-4134-ae47-05e362d22968","Handler":null,"Name":"","Endpoint":""} Oct 30 21:13:17.060860 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:17.060839 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:13:17.060961 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:17.060869 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:13:17.250152 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:17.250114 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p4c9m" event={"ID":"05d96464-f946-4a83-88e5-ececfee30b4c","Type":"ContainerStarted","Data":"5b71cfd9f62d9e37c76025154595d1b368720fde88d3075fe85780d66ddf7b3d"} Oct 30 21:13:17.253417 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:17.253386 2577 generic.go:356] "Generic (PLEG): container finished" podID="25ca3b5e-7eba-4411-be09-5da0fe41e473" containerID="db12be8606824c372929e38943cb47e7ba707b5451d7a7e763d9c780fbc675a1" exitCode=0 Oct 30 21:13:17.253587 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:17.253473 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-brt6h" event={"ID":"25ca3b5e-7eba-4411-be09-5da0fe41e473","Type":"ContainerDied","Data":"db12be8606824c372929e38943cb47e7ba707b5451d7a7e763d9c780fbc675a1"} Oct 30 21:13:17.494804 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:17.494750 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-p4c9m" podStartSLOduration=3.734076967 podStartE2EDuration="12.49473756s" totalImagesPullingTime="8.760660593s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:13:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:17.370302251 +0000 UTC m=+12.807296792" watchObservedRunningTime="2026-10-30 21:13:17.49473756 +0000 UTC m=+12.931732101" Oct 30 21:13:18.117454 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:18.117300 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:18.117454 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:18.117324 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lw64d" Oct 30 21:13:18.117454 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:18.117311 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4g68" Oct 30 21:13:18.117454 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:18.117443 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jztdg" podUID="3341bf47-8c8f-4073-8758-122e04e0b138" Oct 30 21:13:18.118271 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:18.117573 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4g68" podUID="30bde564-b785-4359-9377-ffec880fb18c" Oct 30 21:13:18.118271 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:18.117638 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lw64d" podUID="19538cda-3ba8-4bad-bb47-23a2511736e7" Oct 30 21:13:18.257189 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:18.257154 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" event={"ID":"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8","Type":"ContainerStarted","Data":"f4b7104db2b9e96336ec982fd62ca2ecc6f70c7594a40b8c86b807430feeb075"} Oct 30 21:13:18.258931 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:18.258873 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jfprx" event={"ID":"1d4aae53-afc9-4373-953d-b9b7c323e3ee","Type":"ContainerStarted","Data":"5422e608e692848f5a5a6a395070dc5a4a128cd4efa0291398b5912f41284a7a"} Oct 30 21:13:18.261830 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:18.261799 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" event={"ID":"9597db44-1ca0-4fa8-ade7-cb55eeca9593","Type":"ContainerStarted","Data":"94f05701288ce64132a1b2c6242750a7b4b9e0bba3c4c48083d3f2102a4ce1d9"} Oct 30 21:13:18.493166 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:18.493119 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" podStartSLOduration=3.167760932 podStartE2EDuration="13.493101866s" totalImagesPullingTime="10.325340934s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:13:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:18.37695726 +0000 UTC m=+13.813951792" watchObservedRunningTime="2026-10-30 21:13:18.493101866 +0000 UTC m=+13.930096400" Oct 30 21:13:18.493699 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:18.493666 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jfprx" podStartSLOduration=5.046618203 podStartE2EDuration="13.493656084s" totalImagesPullingTime="8.447037881s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:13:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:18.493197063 +0000 UTC m=+13.930191607" watchObservedRunningTime="2026-10-30 21:13:18.493656084 +0000 UTC m=+13.930650623" Oct 30 21:13:18.612676 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:18.612625 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" podStartSLOduration=3.939331183 podStartE2EDuration="13.612611766s" totalImagesPullingTime="9.673280583s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:13:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:18.612526817 +0000 UTC m=+14.049521358" watchObservedRunningTime="2026-10-30 21:13:18.612611766 +0000 UTC m=+14.049606306" Oct 30 21:13:19.266788 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:19.266755 2577 generic.go:356] "Generic (PLEG): container finished" podID="25ca3b5e-7eba-4411-be09-5da0fe41e473" containerID="ca0c856cf5a71b0d0dffe7d1a48d5b3f4015b2350ae5c82fe0f7e172d80c1084" exitCode=0 Oct 30 21:13:19.267395 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:19.266861 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-brt6h" event={"ID":"25ca3b5e-7eba-4411-be09-5da0fe41e473","Type":"ContainerDied","Data":"ca0c856cf5a71b0d0dffe7d1a48d5b3f4015b2350ae5c82fe0f7e172d80c1084"} Oct 30 21:13:20.117515 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:20.117482 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lw64d" Oct 30 21:13:20.117671 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:20.117560 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4g68" Oct 30 21:13:20.117671 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:20.117669 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:20.117805 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:20.117685 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lw64d" podUID="19538cda-3ba8-4bad-bb47-23a2511736e7" Oct 30 21:13:20.117805 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:20.117755 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jztdg" podUID="3341bf47-8c8f-4073-8758-122e04e0b138" Oct 30 21:13:20.117897 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:20.117821 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4g68" podUID="30bde564-b785-4359-9377-ffec880fb18c" Oct 30 21:13:21.273859 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:21.273652 2577 generic.go:356] "Generic (PLEG): container finished" podID="25ca3b5e-7eba-4411-be09-5da0fe41e473" containerID="d5ec71db208910cf6d3cd5435aa31d642c799a5bf099d6db2e55e4cca269192c" exitCode=0 Oct 30 21:13:21.273859 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:21.273732 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-brt6h" event={"ID":"25ca3b5e-7eba-4411-be09-5da0fe41e473","Type":"ContainerDied","Data":"d5ec71db208910cf6d3cd5435aa31d642c799a5bf099d6db2e55e4cca269192c"} Oct 30 21:13:22.015604 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:22.015581 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30bde564-b785-4359-9377-ffec880fb18c-metrics-certs\") pod \"network-metrics-daemon-n4g68\" (UID: \"30bde564-b785-4359-9377-ffec880fb18c\") " pod="openshift-multus/network-metrics-daemon-n4g68" Oct 30 21:13:22.015685 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:22.015621 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3341bf47-8c8f-4073-8758-122e04e0b138-original-pull-secret\") pod \"global-pull-secret-syncer-jztdg\" (UID: \"3341bf47-8c8f-4073-8758-122e04e0b138\") " pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:22.015731 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:22.015717 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:13:22.015786 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:22.015775 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3341bf47-8c8f-4073-8758-122e04e0b138-original-pull-secret podName:3341bf47-8c8f-4073-8758-122e04e0b138 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:38.015762346 +0000 UTC m=+33.452756864 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3341bf47-8c8f-4073-8758-122e04e0b138-original-pull-secret") pod "global-pull-secret-syncer-jztdg" (UID: "3341bf47-8c8f-4073-8758-122e04e0b138") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:13:22.015830 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:22.015717 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:13:22.015830 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:22.015813 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30bde564-b785-4359-9377-ffec880fb18c-metrics-certs podName:30bde564-b785-4359-9377-ffec880fb18c nodeName:}" failed. No retries permitted until 2026-10-30 21:13:38.015807269 +0000 UTC m=+33.452801787 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30bde564-b785-4359-9377-ffec880fb18c-metrics-certs") pod "network-metrics-daemon-n4g68" (UID: "30bde564-b785-4359-9377-ffec880fb18c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:13:22.116153 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:22.115929 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsrnm\" (UniqueName: \"kubernetes.io/projected/19538cda-3ba8-4bad-bb47-23a2511736e7-kube-api-access-bsrnm\") pod \"network-check-target-lw64d\" (UID: \"19538cda-3ba8-4bad-bb47-23a2511736e7\") " pod="openshift-network-diagnostics/network-check-target-lw64d" Oct 30 21:13:22.116212 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:22.116104 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:13:22.116288 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:22.116230 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:13:22.116288 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:22.116245 2577 projected.go:196] Error preparing data for projected volume kube-api-access-bsrnm for pod openshift-network-diagnostics/network-check-target-lw64d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:13:22.116385 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:22.116296 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19538cda-3ba8-4bad-bb47-23a2511736e7-kube-api-access-bsrnm podName:19538cda-3ba8-4bad-bb47-23a2511736e7 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:38.116279628 +0000 UTC m=+33.553274146 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bsrnm" (UniqueName: "kubernetes.io/projected/19538cda-3ba8-4bad-bb47-23a2511736e7-kube-api-access-bsrnm") pod "network-check-target-lw64d" (UID: "19538cda-3ba8-4bad-bb47-23a2511736e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:13:22.117160 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:22.117143 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lw64d" Oct 30 21:13:22.117259 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:22.117245 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lw64d" podUID="19538cda-3ba8-4bad-bb47-23a2511736e7" Oct 30 21:13:22.117305 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:22.117282 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:22.117356 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:22.117314 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4g68" Oct 30 21:13:22.117440 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:22.117419 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jztdg" podUID="3341bf47-8c8f-4073-8758-122e04e0b138" Oct 30 21:13:22.117533 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:22.117517 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4g68" podUID="30bde564-b785-4359-9377-ffec880fb18c" Oct 30 21:13:22.281772 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:22.281740 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" event={"ID":"18d4542b-53c1-473f-9605-c82fd5a642f4","Type":"ContainerStarted","Data":"dbc0049ff4fc752800f03d77f21f9bc426ba46c6c6a02375e1cde4cf2070cd51"} Oct 30 21:13:22.282119 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:22.281825 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" event={"ID":"18d4542b-53c1-473f-9605-c82fd5a642f4","Type":"ContainerStarted","Data":"933a28dc07d85fe32211ad009d040fc3d75c1b7379ccecaea956dab0b0e47a59"} Oct 30 21:13:22.282119 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:22.281855 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" event={"ID":"18d4542b-53c1-473f-9605-c82fd5a642f4","Type":"ContainerStarted","Data":"2a6ac852831e471729ab5434dddf0e6c5bcf3ac7804906527d482d1a4586654a"} Oct 30 21:13:22.282119 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:22.281880 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" event={"ID":"18d4542b-53c1-473f-9605-c82fd5a642f4","Type":"ContainerStarted","Data":"b0ff3c4b883246787977f49f8a513a7e9d2ab2412568a423916d04bfea552637"} Oct 30 21:13:22.282119 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:22.281906 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" event={"ID":"18d4542b-53c1-473f-9605-c82fd5a642f4","Type":"ContainerStarted","Data":"955a0b8358a8940c1e01aa354cb00c62a5629d2c76e1855b51242b9a17294a2e"} Oct 30 21:13:22.282119 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:22.281931 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" event={"ID":"18d4542b-53c1-473f-9605-c82fd5a642f4","Type":"ContainerStarted","Data":"21c1a0ece08d0444e275e2664fc9a63ecccaca8daf24193d855d36d8a78bd7b2"} Oct 30 21:13:24.117525 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:24.117494 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4g68" Oct 30 21:13:24.117525 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:24.117519 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:24.117989 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:24.117592 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4g68" podUID="30bde564-b785-4359-9377-ffec880fb18c" Oct 30 21:13:24.117989 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:24.117646 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lw64d" Oct 30 21:13:24.117989 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:24.117720 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jztdg" podUID="3341bf47-8c8f-4073-8758-122e04e0b138" Oct 30 21:13:24.117989 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:24.117780 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lw64d" podUID="19538cda-3ba8-4bad-bb47-23a2511736e7" Oct 30 21:13:24.290664 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:24.290638 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" event={"ID":"18d4542b-53c1-473f-9605-c82fd5a642f4","Type":"ContainerStarted","Data":"25c7c27b56a62d2bee1c056ed03353f782bc18ca40def4265d02954eff91330a"} Oct 30 21:13:24.293271 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:24.293249 2577 generic.go:356] "Generic (PLEG): container finished" podID="25ca3b5e-7eba-4411-be09-5da0fe41e473" containerID="9a7acce348aa6e445a497c0abb97cbfa12fd210b0f1492b8a29eabc8869c4a5a" exitCode=0 Oct 30 21:13:24.293363 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:24.293289 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-brt6h" event={"ID":"25ca3b5e-7eba-4411-be09-5da0fe41e473","Type":"ContainerDied","Data":"9a7acce348aa6e445a497c0abb97cbfa12fd210b0f1492b8a29eabc8869c4a5a"} Oct 30 21:13:25.297507 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:25.297469 2577 generic.go:356] "Generic (PLEG): container finished" podID="25ca3b5e-7eba-4411-be09-5da0fe41e473" containerID="0d5063d0b3ed59c57bcba7cc2703a10e8b1c74c180897339c5b7a92896f5fff3" exitCode=0 Oct 30 21:13:25.297979 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:25.297531 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-brt6h" event={"ID":"25ca3b5e-7eba-4411-be09-5da0fe41e473","Type":"ContainerDied","Data":"0d5063d0b3ed59c57bcba7cc2703a10e8b1c74c180897339c5b7a92896f5fff3"} Oct 30 21:13:26.117447 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:26.117421 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4g68" Oct 30 21:13:26.117612 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:26.117457 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lw64d" Oct 30 21:13:26.117612 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:26.117541 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:26.117612 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:26.117553 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4g68" podUID="30bde564-b785-4359-9377-ffec880fb18c" Oct 30 21:13:26.117703 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:26.117620 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lw64d" podUID="19538cda-3ba8-4bad-bb47-23a2511736e7" Oct 30 21:13:26.117703 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:26.117677 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jztdg" podUID="3341bf47-8c8f-4073-8758-122e04e0b138" Oct 30 21:13:26.302527 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:26.302491 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-brt6h" event={"ID":"25ca3b5e-7eba-4411-be09-5da0fe41e473","Type":"ContainerStarted","Data":"08e6fa98981a2d849480a28c63bc552c6625cc8aa73416b412048f27136604e5"} Oct 30 21:13:26.428054 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:26.428004 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-brt6h" podStartSLOduration=8.804690829 podStartE2EDuration="21.427989428s" totalImagesPullingTime="12.623298599s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:13:05 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:26.427538754 +0000 UTC m=+21.864533294" watchObservedRunningTime="2026-10-30 21:13:26.427989428 +0000 UTC m=+21.864983967" Oct 30 21:13:27.308085 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:27.307903 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" event={"ID":"18d4542b-53c1-473f-9605-c82fd5a642f4","Type":"ContainerStarted","Data":"a67e0150ff06e7da7fdcacb6814bd2972a0c26912d7bf9a9dd63821d625eb5ef"} Oct 30 21:13:27.308639 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:27.308433 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:27.308639 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:27.308507 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:27.324775 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:27.324747 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:27.589063 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:27.588944 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" podStartSLOduration=8.33177581 podStartE2EDuration="22.588931935s" totalImagesPullingTime="14.257156125s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:13:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:27.538273248 +0000 UTC m=+22.975267787" watchObservedRunningTime="2026-10-30 21:13:27.588931935 +0000 UTC m=+23.025926474" Oct 30 21:13:28.116633 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:28.116601 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:28.116810 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:28.116677 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lw64d" Oct 30 21:13:28.116810 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:28.116777 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4g68" Oct 30 21:13:28.116810 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:28.116780 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jztdg" podUID="3341bf47-8c8f-4073-8758-122e04e0b138" Oct 30 21:13:28.116944 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:28.116859 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lw64d" podUID="19538cda-3ba8-4bad-bb47-23a2511736e7" Oct 30 21:13:28.116974 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:28.116949 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4g68" podUID="30bde564-b785-4359-9377-ffec880fb18c" Oct 30 21:13:28.310969 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:28.310939 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:28.325862 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:28.325837 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:29.148093 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:29.148008 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jztdg"] Oct 30 21:13:29.148282 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:29.148119 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:29.148282 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:29.148203 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jztdg" podUID="3341bf47-8c8f-4073-8758-122e04e0b138" Oct 30 21:13:29.151616 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:29.151595 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lw64d"] Oct 30 21:13:29.151713 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:29.151667 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lw64d" Oct 30 21:13:29.151758 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:29.151738 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lw64d" podUID="19538cda-3ba8-4bad-bb47-23a2511736e7" Oct 30 21:13:29.152806 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:29.152778 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n4g68"] Oct 30 21:13:29.152962 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:29.152947 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4g68" Oct 30 21:13:29.153107 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:29.153081 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4g68" podUID="30bde564-b785-4359-9377-ffec880fb18c" Oct 30 21:13:31.117093 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:31.117063 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:31.117093 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:31.117091 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4g68" Oct 30 21:13:31.117610 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:31.117094 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lw64d" Oct 30 21:13:31.117610 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:31.117169 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jztdg" podUID="3341bf47-8c8f-4073-8758-122e04e0b138" Oct 30 21:13:31.117610 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:31.117291 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4g68" podUID="30bde564-b785-4359-9377-ffec880fb18c" Oct 30 21:13:31.117610 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:31.117368 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lw64d" podUID="19538cda-3ba8-4bad-bb47-23a2511736e7" Oct 30 21:13:33.117243 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:33.117210 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lw64d" Oct 30 21:13:33.117243 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:33.117234 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:33.117814 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:33.117327 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lw64d" podUID="19538cda-3ba8-4bad-bb47-23a2511736e7" Oct 30 21:13:33.117814 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:33.117395 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jztdg" podUID="3341bf47-8c8f-4073-8758-122e04e0b138" Oct 30 21:13:33.117814 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:33.117402 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4g68" Oct 30 21:13:33.117814 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:33.117502 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4g68" podUID="30bde564-b785-4359-9377-ffec880fb18c" Oct 30 21:13:34.318314 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:34.318086 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeReady" Oct 30 21:13:34.318693 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:34.318378 2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 21:13:34.433424 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:34.433402 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2kqn4"] Oct 30 21:13:34.448900 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:34.448878 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-spbvd"] Oct 30 21:13:34.448998 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:34.448880 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2kqn4" Oct 30 21:13:34.459537 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:34.459520 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-spbvd" Oct 30 21:13:34.503081 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:34.503061 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:13:34.503179 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:34.503105 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:13:34.503179 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:34.503135 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-lm9pr\"" Oct 30 21:13:34.503249 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:34.503180 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:13:34.512518 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:34.512502 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hwnd5\"" Oct 30 21:13:34.534642 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:34.534617 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2kqn4"] Oct 30 21:13:34.539178 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:34.539161 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-spbvd"] Oct 30 21:13:34.603351 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:34.603277 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:13:34.603351 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:34.603305 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:13:34.607528 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:34.607513 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4eed118d-1567-4be4-8c90-5dc0e7478827-metrics-tls\") pod \"dns-default-spbvd\" (UID: \"4eed118d-1567-4be4-8c90-5dc0e7478827\") " pod="openshift-dns/dns-default-spbvd" Oct 30 21:13:34.607586 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:34.607538 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4eed118d-1567-4be4-8c90-5dc0e7478827-config-volume\") pod \"dns-default-spbvd\" (UID: \"4eed118d-1567-4be4-8c90-5dc0e7478827\") " pod="openshift-dns/dns-default-spbvd" Oct 30 21:13:34.607586 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:34.607555 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk4wl\" (UniqueName: \"kubernetes.io/projected/4eed118d-1567-4be4-8c90-5dc0e7478827-kube-api-access-tk4wl\") pod \"dns-default-spbvd\" (UID: \"4eed118d-1567-4be4-8c90-5dc0e7478827\") " pod="openshift-dns/dns-default-spbvd" Oct 30 21:13:34.607586 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:34.607572 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbrm8\" (UniqueName: \"kubernetes.io/projected/3d528ee4-1b51-4fdb-9166-56f14060d6af-kube-api-access-kbrm8\") pod \"ingress-canary-2kqn4\" (UID: \"3d528ee4-1b51-4fdb-9166-56f14060d6af\") " pod="openshift-ingress-canary/ingress-canary-2kqn4" Oct 30 21:13:34.607696 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:34.607589 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d528ee4-1b51-4fdb-9166-56f14060d6af-cert\") pod \"ingress-canary-2kqn4\" (UID: \"3d528ee4-1b51-4fdb-9166-56f14060d6af\") " pod="openshift-ingress-canary/ingress-canary-2kqn4" Oct 30 21:13:34.607696 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:34.607614 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eed118d-1567-4be4-8c90-5dc0e7478827-tmp-dir\") pod \"dns-default-spbvd\" (UID: \"4eed118d-1567-4be4-8c90-5dc0e7478827\") " pod="openshift-dns/dns-default-spbvd" Oct 30 21:13:34.708745 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:34.708716 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4eed118d-1567-4be4-8c90-5dc0e7478827-metrics-tls\") pod \"dns-default-spbvd\" (UID: \"4eed118d-1567-4be4-8c90-5dc0e7478827\") " pod="openshift-dns/dns-default-spbvd" Oct 30 21:13:34.708877 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:34.708753 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4eed118d-1567-4be4-8c90-5dc0e7478827-config-volume\") pod \"dns-default-spbvd\" (UID: \"4eed118d-1567-4be4-8c90-5dc0e7478827\") " pod="openshift-dns/dns-default-spbvd" Oct 30 21:13:34.708877 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:34.708776 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk4wl\" (UniqueName: \"kubernetes.io/projected/4eed118d-1567-4be4-8c90-5dc0e7478827-kube-api-access-tk4wl\") pod \"dns-default-spbvd\" (UID: \"4eed118d-1567-4be4-8c90-5dc0e7478827\") " pod="openshift-dns/dns-default-spbvd" Oct 30 21:13:34.708877 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:34.708804 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbrm8\" (UniqueName: \"kubernetes.io/projected/3d528ee4-1b51-4fdb-9166-56f14060d6af-kube-api-access-kbrm8\") pod \"ingress-canary-2kqn4\" (UID: \"3d528ee4-1b51-4fdb-9166-56f14060d6af\") " pod="openshift-ingress-canary/ingress-canary-2kqn4" Oct 30 21:13:34.708877 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:34.708828 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d528ee4-1b51-4fdb-9166-56f14060d6af-cert\") pod \"ingress-canary-2kqn4\" (UID: \"3d528ee4-1b51-4fdb-9166-56f14060d6af\") " pod="openshift-ingress-canary/ingress-canary-2kqn4" Oct 30 21:13:34.708877 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:34.708856 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eed118d-1567-4be4-8c90-5dc0e7478827-tmp-dir\") pod \"dns-default-spbvd\" (UID: \"4eed118d-1567-4be4-8c90-5dc0e7478827\") " pod="openshift-dns/dns-default-spbvd" Oct 30 21:13:34.709119 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:34.708933 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:13:34.709119 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:34.709006 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4eed118d-1567-4be4-8c90-5dc0e7478827-metrics-tls podName:4eed118d-1567-4be4-8c90-5dc0e7478827 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:35.208990345 +0000 UTC m=+30.645984877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4eed118d-1567-4be4-8c90-5dc0e7478827-metrics-tls") pod "dns-default-spbvd" (UID: "4eed118d-1567-4be4-8c90-5dc0e7478827") : secret "dns-default-metrics-tls" not found Oct 30 21:13:34.709243 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:34.709197 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eed118d-1567-4be4-8c90-5dc0e7478827-tmp-dir\") pod \"dns-default-spbvd\" (UID: \"4eed118d-1567-4be4-8c90-5dc0e7478827\") " pod="openshift-dns/dns-default-spbvd" Oct 30 21:13:34.709432 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:34.709413 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4eed118d-1567-4be4-8c90-5dc0e7478827-config-volume\") pod \"dns-default-spbvd\" (UID: \"4eed118d-1567-4be4-8c90-5dc0e7478827\") " pod="openshift-dns/dns-default-spbvd" Oct 30 21:13:34.712834 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:34.712818 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d528ee4-1b51-4fdb-9166-56f14060d6af-cert\") pod \"ingress-canary-2kqn4\" (UID: \"3d528ee4-1b51-4fdb-9166-56f14060d6af\") " pod="openshift-ingress-canary/ingress-canary-2kqn4" Oct 30 21:13:34.770847 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:34.770819 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk4wl\" (UniqueName: \"kubernetes.io/projected/4eed118d-1567-4be4-8c90-5dc0e7478827-kube-api-access-tk4wl\") pod \"dns-default-spbvd\" (UID: \"4eed118d-1567-4be4-8c90-5dc0e7478827\") " pod="openshift-dns/dns-default-spbvd" Oct 30 21:13:34.771056 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:34.771037 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbrm8\" (UniqueName: \"kubernetes.io/projected/3d528ee4-1b51-4fdb-9166-56f14060d6af-kube-api-access-kbrm8\") pod \"ingress-canary-2kqn4\" (UID: \"3d528ee4-1b51-4fdb-9166-56f14060d6af\") " pod="openshift-ingress-canary/ingress-canary-2kqn4" Oct 30 21:13:35.068292 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.068267 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2kqn4" Oct 30 21:13:35.117479 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.117459 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lw64d" Oct 30 21:13:35.117668 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.117652 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:35.117980 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.117964 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4g68" Oct 30 21:13:35.171655 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.171602 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:13:35.171655 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.171616 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-5fn5j\"" Oct 30 21:13:35.171873 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.171742 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-q4s27\"" Oct 30 21:13:35.171873 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.171742 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w5d85\"" Oct 30 21:13:35.171873 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.171804 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:13:35.172089 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.172071 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:13:35.172126 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.172074 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:13:35.212509 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.212479 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4eed118d-1567-4be4-8c90-5dc0e7478827-metrics-tls\") pod \"dns-default-spbvd\" (UID: \"4eed118d-1567-4be4-8c90-5dc0e7478827\") " pod="openshift-dns/dns-default-spbvd" Oct 30 21:13:35.212623 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:35.212598 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:13:35.212679 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:35.212666 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4eed118d-1567-4be4-8c90-5dc0e7478827-metrics-tls podName:4eed118d-1567-4be4-8c90-5dc0e7478827 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:36.21264781 +0000 UTC m=+31.649642342 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4eed118d-1567-4be4-8c90-5dc0e7478827-metrics-tls") pod "dns-default-spbvd" (UID: "4eed118d-1567-4be4-8c90-5dc0e7478827") : secret "dns-default-metrics-tls" not found Oct 30 21:13:35.316685 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.316657 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hq4sb"] Oct 30 21:13:35.329161 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.329111 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hq4sb" Oct 30 21:13:35.379580 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.379558 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hq4sb"] Oct 30 21:13:35.390581 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.390546 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:13:35.390581 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.390560 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:13:35.390581 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.390574 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b9xqh\"" Oct 30 21:13:35.390746 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.390609 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:13:35.390994 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.390979 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:13:35.414112 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.414092 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17696d35-744c-434c-94de-33ec719feac9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hq4sb\" (UID: \"17696d35-744c-434c-94de-33ec719feac9\") " pod="openshift-insights/insights-runtime-extractor-hq4sb" Oct 30 21:13:35.414186 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.414121 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/17696d35-744c-434c-94de-33ec719feac9-crio-socket\") pod \"insights-runtime-extractor-hq4sb\" (UID: \"17696d35-744c-434c-94de-33ec719feac9\") " pod="openshift-insights/insights-runtime-extractor-hq4sb" Oct 30 21:13:35.414186 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.414160 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17696d35-744c-434c-94de-33ec719feac9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hq4sb\" (UID: \"17696d35-744c-434c-94de-33ec719feac9\") " pod="openshift-insights/insights-runtime-extractor-hq4sb" Oct 30 21:13:35.414302 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.414280 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/17696d35-744c-434c-94de-33ec719feac9-data-volume\") pod \"insights-runtime-extractor-hq4sb\" (UID: \"17696d35-744c-434c-94de-33ec719feac9\") " pod="openshift-insights/insights-runtime-extractor-hq4sb" Oct 30 21:13:35.414379 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.414352 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ghv7\" (UniqueName: \"kubernetes.io/projected/17696d35-744c-434c-94de-33ec719feac9-kube-api-access-5ghv7\") pod \"insights-runtime-extractor-hq4sb\" (UID: \"17696d35-744c-434c-94de-33ec719feac9\") " pod="openshift-insights/insights-runtime-extractor-hq4sb" Oct 30 21:13:35.501252 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.501226 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2kqn4"] Oct 30 21:13:35.504363 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:35.504326 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d528ee4_1b51_4fdb_9166_56f14060d6af.slice/crio-81ee5ef9d70c948a11cb050642ec76add87d6871d46aa1445bd0a65c35fc9d5d WatchSource:0}: Error finding container 81ee5ef9d70c948a11cb050642ec76add87d6871d46aa1445bd0a65c35fc9d5d: Status 404 returned error can't find the container with id 81ee5ef9d70c948a11cb050642ec76add87d6871d46aa1445bd0a65c35fc9d5d Oct 30 21:13:35.514998 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.514978 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17696d35-744c-434c-94de-33ec719feac9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hq4sb\" (UID: \"17696d35-744c-434c-94de-33ec719feac9\") " pod="openshift-insights/insights-runtime-extractor-hq4sb" Oct 30 21:13:35.515131 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.515007 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/17696d35-744c-434c-94de-33ec719feac9-crio-socket\") pod \"insights-runtime-extractor-hq4sb\" (UID: \"17696d35-744c-434c-94de-33ec719feac9\") " pod="openshift-insights/insights-runtime-extractor-hq4sb" Oct 30 21:13:35.515131 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.515034 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17696d35-744c-434c-94de-33ec719feac9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hq4sb\" (UID: \"17696d35-744c-434c-94de-33ec719feac9\") " pod="openshift-insights/insights-runtime-extractor-hq4sb" Oct 30 21:13:35.515131 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.515068 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/17696d35-744c-434c-94de-33ec719feac9-data-volume\") pod \"insights-runtime-extractor-hq4sb\" (UID: \"17696d35-744c-434c-94de-33ec719feac9\") " pod="openshift-insights/insights-runtime-extractor-hq4sb" Oct 30 21:13:35.515131 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:35.515106 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:13:35.515310 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:35.515165 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17696d35-744c-434c-94de-33ec719feac9-kube-rbac-proxy-cm podName:17696d35-744c-434c-94de-33ec719feac9 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:36.015146973 +0000 UTC m=+31.452141511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/17696d35-744c-434c-94de-33ec719feac9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hq4sb" (UID: "17696d35-744c-434c-94de-33ec719feac9") : configmap "kube-rbac-proxy" not found Oct 30 21:13:35.515310 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.515179 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/17696d35-744c-434c-94de-33ec719feac9-crio-socket\") pod \"insights-runtime-extractor-hq4sb\" (UID: \"17696d35-744c-434c-94de-33ec719feac9\") " pod="openshift-insights/insights-runtime-extractor-hq4sb" Oct 30 21:13:35.515310 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.515194 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ghv7\" (UniqueName: \"kubernetes.io/projected/17696d35-744c-434c-94de-33ec719feac9-kube-api-access-5ghv7\") pod \"insights-runtime-extractor-hq4sb\" (UID: \"17696d35-744c-434c-94de-33ec719feac9\") " pod="openshift-insights/insights-runtime-extractor-hq4sb" Oct 30 21:13:35.515310 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:35.515213 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:13:35.515310 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:35.515269 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17696d35-744c-434c-94de-33ec719feac9-insights-runtime-extractor-tls podName:17696d35-744c-434c-94de-33ec719feac9 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:36.015251076 +0000 UTC m=+31.452245597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/17696d35-744c-434c-94de-33ec719feac9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hq4sb" (UID: "17696d35-744c-434c-94de-33ec719feac9") : secret "insights-runtime-extractor-tls" not found Oct 30 21:13:35.515310 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.515282 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/17696d35-744c-434c-94de-33ec719feac9-data-volume\") pod \"insights-runtime-extractor-hq4sb\" (UID: \"17696d35-744c-434c-94de-33ec719feac9\") " pod="openshift-insights/insights-runtime-extractor-hq4sb" Oct 30 21:13:35.578031 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.578007 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ghv7\" (UniqueName: \"kubernetes.io/projected/17696d35-744c-434c-94de-33ec719feac9-kube-api-access-5ghv7\") pod \"insights-runtime-extractor-hq4sb\" (UID: \"17696d35-744c-434c-94de-33ec719feac9\") " pod="openshift-insights/insights-runtime-extractor-hq4sb" Oct 30 21:13:35.615106 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.615052 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-89mnx-pncws"] Oct 30 21:13:35.649454 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.649433 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-89mnx-pncws" Oct 30 21:13:35.673938 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.673912 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-89mnx-pncws"] Oct 30 21:13:35.715953 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.715930 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f285a424-0cc2-41b8-99b9-92db326ea01d-service-ca-bundle\") pod \"periodic-gathering-89mnx-pncws\" (UID: \"f285a424-0cc2-41b8-99b9-92db326ea01d\") " pod="openshift-insights/periodic-gathering-89mnx-pncws" Oct 30 21:13:35.716051 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.715972 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl5cb\" (UniqueName: \"kubernetes.io/projected/f285a424-0cc2-41b8-99b9-92db326ea01d-kube-api-access-gl5cb\") pod \"periodic-gathering-89mnx-pncws\" (UID: \"f285a424-0cc2-41b8-99b9-92db326ea01d\") " pod="openshift-insights/periodic-gathering-89mnx-pncws" Oct 30 21:13:35.716051 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.716019 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f285a424-0cc2-41b8-99b9-92db326ea01d-archives-path\") pod \"periodic-gathering-89mnx-pncws\" (UID: \"f285a424-0cc2-41b8-99b9-92db326ea01d\") " pod="openshift-insights/periodic-gathering-89mnx-pncws" Oct 30 21:13:35.716155 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.716127 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f285a424-0cc2-41b8-99b9-92db326ea01d-trusted-ca-bundle\") pod \"periodic-gathering-89mnx-pncws\" (UID: \"f285a424-0cc2-41b8-99b9-92db326ea01d\") " pod="openshift-insights/periodic-gathering-89mnx-pncws" Oct 30 21:13:35.747957 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.747936 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 21:13:35.799265 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.799243 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-nvvjf\"" Oct 30 21:13:35.799985 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.799969 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 21:13:35.817029 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.817008 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl5cb\" (UniqueName: \"kubernetes.io/projected/f285a424-0cc2-41b8-99b9-92db326ea01d-kube-api-access-gl5cb\") pod \"periodic-gathering-89mnx-pncws\" (UID: \"f285a424-0cc2-41b8-99b9-92db326ea01d\") " pod="openshift-insights/periodic-gathering-89mnx-pncws" Oct 30 21:13:35.817094 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.817064 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f285a424-0cc2-41b8-99b9-92db326ea01d-archives-path\") pod \"periodic-gathering-89mnx-pncws\" (UID: \"f285a424-0cc2-41b8-99b9-92db326ea01d\") " pod="openshift-insights/periodic-gathering-89mnx-pncws" Oct 30 21:13:35.817094 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.817082 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f285a424-0cc2-41b8-99b9-92db326ea01d-trusted-ca-bundle\") pod \"periodic-gathering-89mnx-pncws\" (UID: \"f285a424-0cc2-41b8-99b9-92db326ea01d\") " pod="openshift-insights/periodic-gathering-89mnx-pncws" Oct 30 21:13:35.817314 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.817294 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f285a424-0cc2-41b8-99b9-92db326ea01d-service-ca-bundle\") pod \"periodic-gathering-89mnx-pncws\" (UID: \"f285a424-0cc2-41b8-99b9-92db326ea01d\") " pod="openshift-insights/periodic-gathering-89mnx-pncws" Oct 30 21:13:35.817479 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.817465 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f285a424-0cc2-41b8-99b9-92db326ea01d-archives-path\") pod \"periodic-gathering-89mnx-pncws\" (UID: \"f285a424-0cc2-41b8-99b9-92db326ea01d\") " pod="openshift-insights/periodic-gathering-89mnx-pncws" Oct 30 21:13:35.817774 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.817759 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f285a424-0cc2-41b8-99b9-92db326ea01d-trusted-ca-bundle\") pod \"periodic-gathering-89mnx-pncws\" (UID: \"f285a424-0cc2-41b8-99b9-92db326ea01d\") " pod="openshift-insights/periodic-gathering-89mnx-pncws" Oct 30 21:13:35.817807 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.817777 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f285a424-0cc2-41b8-99b9-92db326ea01d-service-ca-bundle\") pod \"periodic-gathering-89mnx-pncws\" (UID: \"f285a424-0cc2-41b8-99b9-92db326ea01d\") " pod="openshift-insights/periodic-gathering-89mnx-pncws" Oct 30 21:13:35.882636 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.882580 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl5cb\" (UniqueName: \"kubernetes.io/projected/f285a424-0cc2-41b8-99b9-92db326ea01d-kube-api-access-gl5cb\") pod \"periodic-gathering-89mnx-pncws\" (UID: \"f285a424-0cc2-41b8-99b9-92db326ea01d\") " pod="openshift-insights/periodic-gathering-89mnx-pncws" Oct 30 21:13:35.972099 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:35.972056 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-89mnx-pncws" Oct 30 21:13:36.018141 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:36.018112 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17696d35-744c-434c-94de-33ec719feac9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hq4sb\" (UID: \"17696d35-744c-434c-94de-33ec719feac9\") " pod="openshift-insights/insights-runtime-extractor-hq4sb" Oct 30 21:13:36.018252 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:36.018163 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17696d35-744c-434c-94de-33ec719feac9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hq4sb\" (UID: \"17696d35-744c-434c-94de-33ec719feac9\") " pod="openshift-insights/insights-runtime-extractor-hq4sb" Oct 30 21:13:36.018301 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:36.018263 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:13:36.018379 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:36.018365 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:13:36.018379 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:36.018370 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17696d35-744c-434c-94de-33ec719feac9-kube-rbac-proxy-cm podName:17696d35-744c-434c-94de-33ec719feac9 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:37.018319902 +0000 UTC m=+32.455314423 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/17696d35-744c-434c-94de-33ec719feac9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hq4sb" (UID: "17696d35-744c-434c-94de-33ec719feac9") : configmap "kube-rbac-proxy" not found Oct 30 21:13:36.018462 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:36.018412 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17696d35-744c-434c-94de-33ec719feac9-insights-runtime-extractor-tls podName:17696d35-744c-434c-94de-33ec719feac9 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:37.01840156 +0000 UTC m=+32.455396103 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/17696d35-744c-434c-94de-33ec719feac9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hq4sb" (UID: "17696d35-744c-434c-94de-33ec719feac9") : secret "insights-runtime-extractor-tls" not found Oct 30 21:13:36.220760 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:36.220492 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4eed118d-1567-4be4-8c90-5dc0e7478827-metrics-tls\") pod \"dns-default-spbvd\" (UID: \"4eed118d-1567-4be4-8c90-5dc0e7478827\") " pod="openshift-dns/dns-default-spbvd" Oct 30 21:13:36.220916 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:36.220645 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:13:36.220916 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:36.220823 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4eed118d-1567-4be4-8c90-5dc0e7478827-metrics-tls podName:4eed118d-1567-4be4-8c90-5dc0e7478827 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:38.220808543 +0000 UTC m=+33.657803061 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4eed118d-1567-4be4-8c90-5dc0e7478827-metrics-tls") pod "dns-default-spbvd" (UID: "4eed118d-1567-4be4-8c90-5dc0e7478827") : secret "dns-default-metrics-tls" not found Oct 30 21:13:36.326406 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:36.326374 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2kqn4" event={"ID":"3d528ee4-1b51-4fdb-9166-56f14060d6af","Type":"ContainerStarted","Data":"81ee5ef9d70c948a11cb050642ec76add87d6871d46aa1445bd0a65c35fc9d5d"} Oct 30 21:13:36.363978 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:36.363422 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-89mnx-pncws"] Oct 30 21:13:36.366276 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:36.366250 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf285a424_0cc2_41b8_99b9_92db326ea01d.slice/crio-8312a77f41c875aae9eea8ead18164c2574c4b690333c9efaae39ac09e4f37e0 WatchSource:0}: Error finding container 8312a77f41c875aae9eea8ead18164c2574c4b690333c9efaae39ac09e4f37e0: Status 404 returned error can't find the container with id 8312a77f41c875aae9eea8ead18164c2574c4b690333c9efaae39ac09e4f37e0 Oct 30 21:13:36.955949 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:36.955867 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-hz5dc"] Oct 30 21:13:36.959621 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:36.959586 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hz5dc" Oct 30 21:13:37.004041 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:37.004008 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-zxlqv"] Oct 30 21:13:37.006351 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:37.006321 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-zxlqv" Oct 30 21:13:37.018017 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:37.017996 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 21:13:37.019116 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:37.019096 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-hz5dc"] Oct 30 21:13:37.019325 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:37.019308 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-9zv66\"" Oct 30 21:13:37.019478 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:37.019410 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 21:13:37.028174 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:37.028152 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfs8x\" (UniqueName: \"kubernetes.io/projected/70c3c6bc-c455-4fe6-9099-1ceb92abeaf0-kube-api-access-gfs8x\") pod \"migrator-66df5d496b-hz5dc\" (UID: \"70c3c6bc-c455-4fe6-9099-1ceb92abeaf0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hz5dc" Oct 30 21:13:37.028275 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:37.028210 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17696d35-744c-434c-94de-33ec719feac9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hq4sb\" (UID: \"17696d35-744c-434c-94de-33ec719feac9\") " pod="openshift-insights/insights-runtime-extractor-hq4sb" Oct 30 21:13:37.028405 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:37.028387 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:13:37.028460 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:37.028421 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17696d35-744c-434c-94de-33ec719feac9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hq4sb\" (UID: \"17696d35-744c-434c-94de-33ec719feac9\") " pod="openshift-insights/insights-runtime-extractor-hq4sb" Oct 30 21:13:37.028460 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:37.028449 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17696d35-744c-434c-94de-33ec719feac9-insights-runtime-extractor-tls podName:17696d35-744c-434c-94de-33ec719feac9 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:39.028431774 +0000 UTC m=+34.465426296 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/17696d35-744c-434c-94de-33ec719feac9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hq4sb" (UID: "17696d35-744c-434c-94de-33ec719feac9") : secret "insights-runtime-extractor-tls" not found Oct 30 21:13:37.028578 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:37.028491 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:13:37.028578 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:37.028562 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17696d35-744c-434c-94de-33ec719feac9-kube-rbac-proxy-cm podName:17696d35-744c-434c-94de-33ec719feac9 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:39.028545679 +0000 UTC m=+34.465540211 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/17696d35-744c-434c-94de-33ec719feac9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hq4sb" (UID: "17696d35-744c-434c-94de-33ec719feac9") : configmap "kube-rbac-proxy" not found Oct 30 21:13:37.078175 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:37.078150 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-zxlqv"] Oct 30 21:13:37.117846 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:37.117799 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 21:13:37.118010 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:37.117980 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 21:13:37.118113 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:37.118097 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 21:13:37.118113 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:37.118108 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-2smgj\"" Oct 30 21:13:37.118207 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:37.118147 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 21:13:37.129283 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:37.129257 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f96e4d00-99fb-4196-a1a9-83599bfd875f-signing-cabundle\") pod \"service-ca-5fffc97bdf-zxlqv\" (UID: \"f96e4d00-99fb-4196-a1a9-83599bfd875f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zxlqv" Oct 30 21:13:37.129451 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:37.129294 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdp4m\" (UniqueName: \"kubernetes.io/projected/f96e4d00-99fb-4196-a1a9-83599bfd875f-kube-api-access-kdp4m\") pod \"service-ca-5fffc97bdf-zxlqv\" (UID: \"f96e4d00-99fb-4196-a1a9-83599bfd875f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zxlqv" Oct 30 21:13:37.129515 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:37.129456 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f96e4d00-99fb-4196-a1a9-83599bfd875f-signing-key\") pod \"service-ca-5fffc97bdf-zxlqv\" (UID: \"f96e4d00-99fb-4196-a1a9-83599bfd875f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zxlqv" Oct 30 21:13:37.129515 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:37.129486 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfs8x\" (UniqueName: \"kubernetes.io/projected/70c3c6bc-c455-4fe6-9099-1ceb92abeaf0-kube-api-access-gfs8x\") pod \"migrator-66df5d496b-hz5dc\" (UID: \"70c3c6bc-c455-4fe6-9099-1ceb92abeaf0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hz5dc" Oct 30 21:13:37.192471 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:37.192442 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfs8x\" (UniqueName: \"kubernetes.io/projected/70c3c6bc-c455-4fe6-9099-1ceb92abeaf0-kube-api-access-gfs8x\") pod \"migrator-66df5d496b-hz5dc\" (UID: \"70c3c6bc-c455-4fe6-9099-1ceb92abeaf0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hz5dc" Oct 30 21:13:37.230662 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:37.230628 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f96e4d00-99fb-4196-a1a9-83599bfd875f-signing-cabundle\") pod \"service-ca-5fffc97bdf-zxlqv\" (UID: \"f96e4d00-99fb-4196-a1a9-83599bfd875f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zxlqv" Oct 30 21:13:37.230798 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:37.230670 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdp4m\" (UniqueName: \"kubernetes.io/projected/f96e4d00-99fb-4196-a1a9-83599bfd875f-kube-api-access-kdp4m\") pod \"service-ca-5fffc97bdf-zxlqv\" (UID: \"f96e4d00-99fb-4196-a1a9-83599bfd875f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zxlqv" Oct 30 21:13:37.230798 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:37.230744 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f96e4d00-99fb-4196-a1a9-83599bfd875f-signing-key\") pod \"service-ca-5fffc97bdf-zxlqv\" (UID: \"f96e4d00-99fb-4196-a1a9-83599bfd875f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zxlqv" Oct 30 21:13:37.231358 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:37.231313 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f96e4d00-99fb-4196-a1a9-83599bfd875f-signing-cabundle\") pod \"service-ca-5fffc97bdf-zxlqv\" (UID: \"f96e4d00-99fb-4196-a1a9-83599bfd875f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zxlqv" Oct 30 21:13:37.233292 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:37.233271 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f96e4d00-99fb-4196-a1a9-83599bfd875f-signing-key\") pod \"service-ca-5fffc97bdf-zxlqv\" (UID: \"f96e4d00-99fb-4196-a1a9-83599bfd875f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zxlqv" Oct 30 21:13:37.271565 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:37.271542 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hz5dc" Oct 30 21:13:37.292759 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:37.292737 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdp4m\" (UniqueName: \"kubernetes.io/projected/f96e4d00-99fb-4196-a1a9-83599bfd875f-kube-api-access-kdp4m\") pod \"service-ca-5fffc97bdf-zxlqv\" (UID: \"f96e4d00-99fb-4196-a1a9-83599bfd875f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zxlqv" Oct 30 21:13:37.317184 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:37.316951 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-zxlqv" Oct 30 21:13:37.331825 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:37.331800 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-89mnx-pncws" event={"ID":"f285a424-0cc2-41b8-99b9-92db326ea01d","Type":"ContainerStarted","Data":"8312a77f41c875aae9eea8ead18164c2574c4b690333c9efaae39ac09e4f37e0"} Oct 30 21:13:37.654205 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:37.654099 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-hz5dc"] Oct 30 21:13:37.693264 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:37.693232 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70c3c6bc_c455_4fe6_9099_1ceb92abeaf0.slice/crio-6d6f28503f3ba781641478ee756d05fee509b2b0f70a1825e79bbb2576ee9849 WatchSource:0}: Error finding container 6d6f28503f3ba781641478ee756d05fee509b2b0f70a1825e79bbb2576ee9849: Status 404 returned error can't find the container with id 6d6f28503f3ba781641478ee756d05fee509b2b0f70a1825e79bbb2576ee9849 Oct 30 21:13:37.701582 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:37.701558 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-zxlqv"] Oct 30 21:13:37.704780 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:37.704754 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf96e4d00_99fb_4196_a1a9_83599bfd875f.slice/crio-c2e5dac0e9ba9d2162d1da40dd6bd04b66a2a324a2fd34b57dedc7cb1bcdc7ad WatchSource:0}: Error finding container c2e5dac0e9ba9d2162d1da40dd6bd04b66a2a324a2fd34b57dedc7cb1bcdc7ad: Status 404 returned error can't find the container with id c2e5dac0e9ba9d2162d1da40dd6bd04b66a2a324a2fd34b57dedc7cb1bcdc7ad Oct 30 21:13:38.037143 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:38.036638 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3341bf47-8c8f-4073-8758-122e04e0b138-original-pull-secret\") pod \"global-pull-secret-syncer-jztdg\" (UID: \"3341bf47-8c8f-4073-8758-122e04e0b138\") " pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:38.037143 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:38.036726 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30bde564-b785-4359-9377-ffec880fb18c-metrics-certs\") pod \"network-metrics-daemon-n4g68\" (UID: \"30bde564-b785-4359-9377-ffec880fb18c\") " pod="openshift-multus/network-metrics-daemon-n4g68" Oct 30 21:13:38.037143 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:38.036835 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:13:38.037143 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:38.036901 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30bde564-b785-4359-9377-ffec880fb18c-metrics-certs podName:30bde564-b785-4359-9377-ffec880fb18c nodeName:}" failed. No retries permitted until 2026-10-30 21:14:10.03688306 +0000 UTC m=+65.473877578 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30bde564-b785-4359-9377-ffec880fb18c-metrics-certs") pod "network-metrics-daemon-n4g68" (UID: "30bde564-b785-4359-9377-ffec880fb18c") : secret "metrics-daemon-secret" not found Oct 30 21:13:38.039359 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:38.039323 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3341bf47-8c8f-4073-8758-122e04e0b138-original-pull-secret\") pod \"global-pull-secret-syncer-jztdg\" (UID: \"3341bf47-8c8f-4073-8758-122e04e0b138\") " pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:38.137660 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:38.137618 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsrnm\" (UniqueName: \"kubernetes.io/projected/19538cda-3ba8-4bad-bb47-23a2511736e7-kube-api-access-bsrnm\") pod \"network-check-target-lw64d\" (UID: \"19538cda-3ba8-4bad-bb47-23a2511736e7\") " pod="openshift-network-diagnostics/network-check-target-lw64d" Oct 30 21:13:38.140575 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:38.140551 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsrnm\" (UniqueName: \"kubernetes.io/projected/19538cda-3ba8-4bad-bb47-23a2511736e7-kube-api-access-bsrnm\") pod \"network-check-target-lw64d\" (UID: \"19538cda-3ba8-4bad-bb47-23a2511736e7\") " pod="openshift-network-diagnostics/network-check-target-lw64d" Oct 30 21:13:38.144781 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:38.144542 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lw64d" Oct 30 21:13:38.156648 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:38.156622 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:38.238813 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:38.238785 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4eed118d-1567-4be4-8c90-5dc0e7478827-metrics-tls\") pod \"dns-default-spbvd\" (UID: \"4eed118d-1567-4be4-8c90-5dc0e7478827\") " pod="openshift-dns/dns-default-spbvd" Oct 30 21:13:38.238962 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:38.238948 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:13:38.239024 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:38.239003 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4eed118d-1567-4be4-8c90-5dc0e7478827-metrics-tls podName:4eed118d-1567-4be4-8c90-5dc0e7478827 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:42.238990823 +0000 UTC m=+37.675985355 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4eed118d-1567-4be4-8c90-5dc0e7478827-metrics-tls") pod "dns-default-spbvd" (UID: "4eed118d-1567-4be4-8c90-5dc0e7478827") : secret "dns-default-metrics-tls" not found Oct 30 21:13:38.336078 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:38.335983 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-zxlqv" event={"ID":"f96e4d00-99fb-4196-a1a9-83599bfd875f","Type":"ContainerStarted","Data":"c2e5dac0e9ba9d2162d1da40dd6bd04b66a2a324a2fd34b57dedc7cb1bcdc7ad"} Oct 30 21:13:38.337760 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:38.337730 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hz5dc" event={"ID":"70c3c6bc-c455-4fe6-9099-1ceb92abeaf0","Type":"ContainerStarted","Data":"6d6f28503f3ba781641478ee756d05fee509b2b0f70a1825e79bbb2576ee9849"} Oct 30 21:13:38.526933 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:38.526893 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lw64d"] Oct 30 21:13:38.531769 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:38.531740 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19538cda_3ba8_4bad_bb47_23a2511736e7.slice/crio-5b0979854386d0a46f5c2c6c58f661c5f84538da48cee36194af915536819a42 WatchSource:0}: Error finding container 5b0979854386d0a46f5c2c6c58f661c5f84538da48cee36194af915536819a42: Status 404 returned error can't find the container with id 5b0979854386d0a46f5c2c6c58f661c5f84538da48cee36194af915536819a42 Oct 30 21:13:38.579240 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:38.578164 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jztdg"] Oct 30 21:13:38.586854 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:38.582529 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3341bf47_8c8f_4073_8758_122e04e0b138.slice/crio-821d2084423f5098c7e92e6ab480a1dae74e29ea70ce538ce25533941e5ec63b WatchSource:0}: Error finding container 821d2084423f5098c7e92e6ab480a1dae74e29ea70ce538ce25533941e5ec63b: Status 404 returned error can't find the container with id 821d2084423f5098c7e92e6ab480a1dae74e29ea70ce538ce25533941e5ec63b Oct 30 21:13:39.050873 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:39.050815 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17696d35-744c-434c-94de-33ec719feac9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hq4sb\" (UID: \"17696d35-744c-434c-94de-33ec719feac9\") " pod="openshift-insights/insights-runtime-extractor-hq4sb" Oct 30 21:13:39.051457 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:39.050877 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17696d35-744c-434c-94de-33ec719feac9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hq4sb\" (UID: \"17696d35-744c-434c-94de-33ec719feac9\") " pod="openshift-insights/insights-runtime-extractor-hq4sb" Oct 30 21:13:39.051457 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:39.051046 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:13:39.051457 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:39.051110 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17696d35-744c-434c-94de-33ec719feac9-insights-runtime-extractor-tls podName:17696d35-744c-434c-94de-33ec719feac9 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:43.051092186 +0000 UTC m=+38.488086707 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/17696d35-744c-434c-94de-33ec719feac9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hq4sb" (UID: "17696d35-744c-434c-94de-33ec719feac9") : secret "insights-runtime-extractor-tls" not found Oct 30 21:13:39.051648 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:39.051510 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:13:39.051648 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:39.051563 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17696d35-744c-434c-94de-33ec719feac9-kube-rbac-proxy-cm podName:17696d35-744c-434c-94de-33ec719feac9 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:43.051548235 +0000 UTC m=+38.488542752 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/17696d35-744c-434c-94de-33ec719feac9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hq4sb" (UID: "17696d35-744c-434c-94de-33ec719feac9") : configmap "kube-rbac-proxy" not found Oct 30 21:13:39.343013 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:39.342888 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-89mnx-pncws" event={"ID":"f285a424-0cc2-41b8-99b9-92db326ea01d","Type":"ContainerStarted","Data":"dd90042e5200ab054b9455faf195f596928050d38efdde07a6402d271da4f0bc"} Oct 30 21:13:39.345795 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:39.345764 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2kqn4" event={"ID":"3d528ee4-1b51-4fdb-9166-56f14060d6af","Type":"ContainerStarted","Data":"92aab98cb479b9d610b3cea043615b0b12b4b018e6c39160bbe7f61c6022e0db"} Oct 30 21:13:39.347175 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:39.347145 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lw64d" event={"ID":"19538cda-3ba8-4bad-bb47-23a2511736e7","Type":"ContainerStarted","Data":"5b0979854386d0a46f5c2c6c58f661c5f84538da48cee36194af915536819a42"} Oct 30 21:13:39.348546 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:39.348524 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jztdg" event={"ID":"3341bf47-8c8f-4073-8758-122e04e0b138","Type":"ContainerStarted","Data":"821d2084423f5098c7e92e6ab480a1dae74e29ea70ce538ce25533941e5ec63b"} Oct 30 21:13:39.461488 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:39.461432 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-89mnx-pncws" podStartSLOduration=1.6689710949999998 podStartE2EDuration="4.461414822s" totalImagesPullingTime="2.792443727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:39.461290395 +0000 UTC m=+34.898284935" watchObservedRunningTime="2026-10-30 21:13:39.461414822 +0000 UTC m=+34.898409365" Oct 30 21:13:39.579843 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:39.579773 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2kqn4" podStartSLOduration=2.361925044 podStartE2EDuration="5.579756073s" totalImagesPullingTime="3.217831029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:39.579125709 +0000 UTC m=+35.016120250" watchObservedRunningTime="2026-10-30 21:13:39.579756073 +0000 UTC m=+35.016750612" Oct 30 21:13:40.355043 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:40.355009 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hz5dc" event={"ID":"70c3c6bc-c455-4fe6-9099-1ceb92abeaf0","Type":"ContainerStarted","Data":"029327ad815520dcd62dbf996bba97df40275f0eb3eb4d6c05616fa4d986cbc5"} Oct 30 21:13:40.472585 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:40.472531 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hz5dc" podStartSLOduration=2.68506599 podStartE2EDuration="4.472516013s" totalImagesPullingTime="1.787450023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:40.471372871 +0000 UTC m=+35.908367413" watchObservedRunningTime="2026-10-30 21:13:40.472516013 +0000 UTC m=+35.909510552" Oct 30 21:13:41.358830 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:41.358798 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-zxlqv" event={"ID":"f96e4d00-99fb-4196-a1a9-83599bfd875f","Type":"ContainerStarted","Data":"a4e29ae9a61576d71e926c7e4a697c2d59b20f458d63454d8b432d762e01c785"} Oct 30 21:13:41.477258 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:41.477206 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-zxlqv" podStartSLOduration=2.216967549 podStartE2EDuration="5.477191161s" totalImagesPullingTime="3.260223612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:41.476782106 +0000 UTC m=+36.913776647" watchObservedRunningTime="2026-10-30 21:13:41.477191161 +0000 UTC m=+36.914185702" Oct 30 21:13:42.276646 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:42.276606 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4eed118d-1567-4be4-8c90-5dc0e7478827-metrics-tls\") pod \"dns-default-spbvd\" (UID: \"4eed118d-1567-4be4-8c90-5dc0e7478827\") " pod="openshift-dns/dns-default-spbvd" Oct 30 21:13:42.276834 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:42.276751 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:13:42.276834 ip-10-0-129-83 kubenswrapper[2577]: E1030 21:13:42.276810 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4eed118d-1567-4be4-8c90-5dc0e7478827-metrics-tls podName:4eed118d-1567-4be4-8c90-5dc0e7478827 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:50.276793643 +0000 UTC m=+45.713788167 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4eed118d-1567-4be4-8c90-5dc0e7478827-metrics-tls") pod "dns-default-spbvd" (UID: "4eed118d-1567-4be4-8c90-5dc0e7478827") : secret "dns-default-metrics-tls" not found Oct 30 21:13:42.702787 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:42.702702 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-p4c9m_05d96464-f946-4a83-88e5-ececfee30b4c/dns-node-resolver/0.log" Oct 30 21:13:43.082999 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:43.082961 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17696d35-744c-434c-94de-33ec719feac9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hq4sb\" (UID: \"17696d35-744c-434c-94de-33ec719feac9\") " pod="openshift-insights/insights-runtime-extractor-hq4sb" Oct 30 21:13:43.083179 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:43.083064 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17696d35-744c-434c-94de-33ec719feac9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hq4sb\" (UID: \"17696d35-744c-434c-94de-33ec719feac9\") " pod="openshift-insights/insights-runtime-extractor-hq4sb" Oct 30 21:13:43.083697 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:43.083676 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17696d35-744c-434c-94de-33ec719feac9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hq4sb\" (UID: \"17696d35-744c-434c-94de-33ec719feac9\") " pod="openshift-insights/insights-runtime-extractor-hq4sb" Oct 30 21:13:43.086207 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:43.086185 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17696d35-744c-434c-94de-33ec719feac9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hq4sb\" (UID: \"17696d35-744c-434c-94de-33ec719feac9\") " pod="openshift-insights/insights-runtime-extractor-hq4sb" Oct 30 21:13:43.172220 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:43.172189 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hq4sb" Oct 30 21:13:43.367967 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:43.367883 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lw64d" event={"ID":"19538cda-3ba8-4bad-bb47-23a2511736e7","Type":"ContainerStarted","Data":"1578046e823cc884e911819c0f537d491b44f65bba6339ecf3211b7a4d15df4f"} Oct 30 21:13:43.368129 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:43.368029 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lw64d" Oct 30 21:13:43.486302 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:43.485977 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lw64d" podStartSLOduration=34.450216232 podStartE2EDuration="38.485960178s" totalImagesPullingTime="4.035743946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:43.485806956 +0000 UTC m=+38.922801500" watchObservedRunningTime="2026-10-30 21:13:43.485960178 +0000 UTC m=+38.922954718" Oct 30 21:13:43.657929 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:43.657850 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hq4sb"] Oct 30 21:13:43.661273 ip-10-0-129-83 kubenswrapper[2577]: W1030 21:13:43.661243 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17696d35_744c_434c_94de_33ec719feac9.slice/crio-f929d541705328cc0720bdb06c631f4ff12e9e449eebb887da4e855b09568ec9 WatchSource:0}: Error finding container f929d541705328cc0720bdb06c631f4ff12e9e449eebb887da4e855b09568ec9: Status 404 returned error can't find the container with id f929d541705328cc0720bdb06c631f4ff12e9e449eebb887da4e855b09568ec9 Oct 30 21:13:43.899858 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:43.899834 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q8g7b_54b8c0a5-e1a7-4d02-a402-b3cadc139e28/node-ca/0.log" Oct 30 21:13:44.377090 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:44.377009 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hq4sb" event={"ID":"17696d35-744c-434c-94de-33ec719feac9","Type":"ContainerStarted","Data":"b10549da92338bfda20f47e3ea76062cfdcb5a985707f9dd7c5e27b983888faa"} Oct 30 21:13:44.377242 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:44.377106 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hq4sb" event={"ID":"17696d35-744c-434c-94de-33ec719feac9","Type":"ContainerStarted","Data":"f929d541705328cc0720bdb06c631f4ff12e9e449eebb887da4e855b09568ec9"} Oct 30 21:13:44.378626 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:44.378584 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jztdg" event={"ID":"3341bf47-8c8f-4073-8758-122e04e0b138","Type":"ContainerStarted","Data":"ce1ce3e69581c2082a99f2c4b2642fa0f7b41d1f9bf2aaaa36020985cff45463"} Oct 30 21:13:44.387421 ip-10-0-129-83 kubenswrapper[2577]: I1030 21:13:44.387399 2577 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:13:44.387587 ip-10-0-129-83 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 21:13:44.398689 ip-10-0-129-83 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 21:13:44.398853 ip-10-0-129-83 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 21:13:44.399025 ip-10-0-129-83 systemd[1]: kubelet.service: Consumed 2.371s CPU time. Oct 30 21:13:44.415840 ip-10-0-129-83 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:13:44.474919 ip-10-0-129-83 kubenswrapper[4560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:13:44.474919 ip-10-0-129-83 kubenswrapper[4560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:13:44.474919 ip-10-0-129-83 kubenswrapper[4560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:13:44.474919 ip-10-0-129-83 kubenswrapper[4560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:13:44.476920 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.476849 4560 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:13:44.476920 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.476921 4560 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:13:44.476993 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.476925 4560 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:13:44.476993 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.476928 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:13:44.476993 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.476931 4560 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:13:44.476993 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.476934 4560 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:13:44.476993 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.476937 4560 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:13:44.476993 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.476941 4560 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:13:44.476993 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.476944 4560 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:13:44.476993 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.476947 4560 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:13:44.476993 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.476950 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:13:44.476993 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.476952 4560 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:13:44.476993 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.476955 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:13:44.476993 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.476958 4560 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:13:44.476993 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.476960 4560 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:13:44.476993 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.476963 4560 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:13:44.476993 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.476965 4560 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:13:44.476993 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.476968 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:13:44.476993 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.476970 4560 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:13:44.476993 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.476974 4560 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:13:44.476993 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.476977 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:13:44.477482 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.476979 4560 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:13:44.477482 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.476982 4560 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:13:44.477482 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.476984 4560 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:13:44.477482 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.476987 4560 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:13:44.477482 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.476989 4560 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:13:44.477482 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.476993 4560 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:13:44.477482 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.476998 4560 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:13:44.477482 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477005 4560 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:13:44.477482 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477008 4560 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:13:44.477482 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477010 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:13:44.477482 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477013 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:13:44.477482 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477016 4560 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:13:44.477482 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477018 4560 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:13:44.477482 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477021 4560 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:13:44.477482 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477024 4560 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:13:44.477482 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477026 4560 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:13:44.477482 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477029 4560 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:13:44.477482 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477032 4560 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:13:44.477482 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477034 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:13:44.477482 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477037 4560 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:13:44.477965 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477040 4560 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:13:44.477965 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477042 4560 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:13:44.477965 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477045 4560 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:13:44.477965 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477047 4560 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:13:44.477965 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477050 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:13:44.477965 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477052 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:13:44.477965 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477055 4560 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:13:44.477965 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477058 4560 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:13:44.477965 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477060 4560 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:13:44.477965 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477063 4560 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:13:44.477965 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477065 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:13:44.477965 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477068 4560 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:13:44.477965 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477070 4560 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:13:44.477965 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477074 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:13:44.477965 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477077 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:13:44.477965 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477081 4560 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:13:44.477965 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477084 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:13:44.477965 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477087 4560 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:13:44.477965 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477090 4560 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:13:44.477965 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477093 4560 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:13:44.478508 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477096 4560 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:13:44.478508 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477098 4560 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:13:44.478508 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477101 4560 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:13:44.478508 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477103 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:13:44.478508 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477106 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:13:44.478508 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477109 4560 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:13:44.478508 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477117 4560 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:13:44.478508 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477120 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:13:44.478508 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477123 4560 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:13:44.478508 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477126 4560 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:13:44.478508 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477129 4560 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:13:44.478508 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477131 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:13:44.478508 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477134 4560 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:13:44.478508 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477137 4560 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:13:44.478508 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477139 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:13:44.478508 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477142 4560 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:13:44.478508 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477145 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:13:44.478508 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477148 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:13:44.478508 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477150 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:13:44.478971 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477153 4560 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:13:44.478971 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477155 4560 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:13:44.478971 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477158 4560 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:13:44.478971 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477161 4560 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:13:44.478971 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477163 4560 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:13:44.478971 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477166 4560 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:13:44.478971 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477169 4560 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:13:44.478971 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477171 4560 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:13:44.478971 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477174 4560 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:13:44.478971 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477177 4560 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:13:44.478971 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477180 4560 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:13:44.478971 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477183 4560 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:13:44.478971 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477186 4560 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:13:44.478971 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477188 4560 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:13:44.478971 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477191 4560 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:13:44.478971 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477194 4560 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:13:44.478971 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477197 4560 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:13:44.478971 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477200 4560 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:13:44.478971 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477202 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:13:44.478971 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477205 4560 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:13:44.479502 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477207 4560 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:13:44.479502 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477210 4560 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:13:44.479502 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477213 4560 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:13:44.479502 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477215 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:13:44.479502 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477218 4560 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:13:44.479502 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.477220 4560 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:13:44.479502 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478014 4560 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:13:44.479502 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478021 4560 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:13:44.479502 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478024 4560 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:13:44.479502 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478027 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:13:44.479502 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478030 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:13:44.479502 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478032 4560 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:13:44.479502 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478035 4560 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:13:44.479502 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478038 4560 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:13:44.479502 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478040 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:13:44.479502 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478044 4560 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:13:44.479502 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478047 4560 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:13:44.479502 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478049 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:13:44.479502 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478055 4560 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:13:44.479502 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478057 4560 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:13:44.479994 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478060 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:13:44.479994 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478063 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:13:44.479994 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478073 4560 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:13:44.479994 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478076 4560 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:13:44.479994 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478079 4560 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:13:44.479994 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478082 4560 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:13:44.479994 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478084 4560 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:13:44.479994 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478093 4560 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:13:44.479994 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478096 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:13:44.479994 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478099 4560 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:13:44.479994 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478101 4560 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:13:44.479994 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478104 4560 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:13:44.479994 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478106 4560 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:13:44.479994 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478109 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:13:44.479994 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478111 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:13:44.479994 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478114 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:13:44.479994 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478117 4560 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:13:44.479994 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478119 4560 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:13:44.479994 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478122 4560 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:13:44.480503 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478125 4560 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:13:44.480503 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478128 4560 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:13:44.480503 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478131 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:13:44.480503 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478133 4560 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:13:44.480503 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478136 4560 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:13:44.480503 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478138 4560 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:13:44.480503 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478141 4560 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:13:44.480503 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478143 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:13:44.480503 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478146 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:13:44.480503 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478149 4560 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:13:44.480503 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478151 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:13:44.480503 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478155 4560 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:13:44.480503 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478158 4560 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:13:44.480503 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478160 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:13:44.480503 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478163 4560 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:13:44.480503 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478165 4560 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:13:44.480503 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478168 4560 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:13:44.480503 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478172 4560 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:13:44.480503 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478176 4560 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:13:44.480955 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478180 4560 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:13:44.480955 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478183 4560 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:13:44.480955 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478189 4560 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:13:44.480955 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478192 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:13:44.480955 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478194 4560 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:13:44.480955 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478197 4560 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:13:44.480955 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478199 4560 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:13:44.480955 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478202 4560 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:13:44.480955 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478205 4560 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:13:44.480955 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478207 4560 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:13:44.480955 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478210 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:13:44.480955 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478213 4560 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:13:44.480955 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478215 4560 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:13:44.480955 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478218 4560 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:13:44.480955 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478220 4560 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:13:44.480955 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478223 4560 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:13:44.480955 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478225 4560 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:13:44.480955 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478228 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:13:44.480955 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478230 4560 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:13:44.480955 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478233 4560 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:13:44.481556 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478235 4560 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:13:44.481556 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478238 4560 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:13:44.481556 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478240 4560 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:13:44.481556 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478243 4560 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:13:44.481556 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478247 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:13:44.481556 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478250 4560 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:13:44.481556 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478252 4560 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:13:44.481556 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478255 4560 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:13:44.481556 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478258 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:13:44.481556 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478261 4560 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:13:44.481556 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478263 4560 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:13:44.481556 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478266 4560 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:13:44.481556 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478268 4560 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:13:44.481556 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478271 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:13:44.481556 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478274 4560 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:13:44.481556 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478279 4560 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:13:44.481556 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478282 4560 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:13:44.481556 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478284 4560 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:13:44.481556 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478287 4560 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:13:44.482016 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478290 4560 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:13:44.482016 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478294 4560 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:13:44.482016 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478297 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:13:44.482016 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478301 4560 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:13:44.482016 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478303 4560 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:13:44.482016 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478307 4560 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:13:44.482016 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478309 4560 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:13:44.482016 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478312 4560 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:13:44.482016 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478315 4560 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:13:44.482016 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478318 4560 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:13:44.482016 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478320 4560 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:13:44.482016 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478323 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:13:44.482016 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478325 4560 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:13:44.482016 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478328 4560 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:13:44.482016 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478331 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:13:44.482016 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478812 4560 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:13:44.482016 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478818 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:13:44.482016 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478823 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:13:44.482016 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478826 4560 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:13:44.482491 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478828 4560 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:13:44.482491 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478831 4560 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:13:44.482491 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478834 4560 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:13:44.482491 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478837 4560 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:13:44.482491 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478839 4560 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:13:44.482491 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478842 4560 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:13:44.482491 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478844 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:13:44.482491 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478847 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:13:44.482491 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478849 4560 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:13:44.482491 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478853 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:13:44.482491 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478855 4560 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:13:44.482491 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478858 4560 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:13:44.482491 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478861 4560 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:13:44.482491 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478864 4560 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:13:44.482491 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478867 4560 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:13:44.482491 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478870 4560 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:13:44.482491 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478872 4560 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:13:44.482491 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478875 4560 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:13:44.482491 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478877 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:13:44.482491 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478880 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:13:44.483011 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478883 4560 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:13:44.483011 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478885 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:13:44.483011 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478888 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:13:44.483011 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478890 4560 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:13:44.483011 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478893 4560 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:13:44.483011 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478895 4560 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:13:44.483011 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478898 4560 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:13:44.483011 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478901 4560 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:13:44.483011 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478903 4560 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:13:44.483011 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478906 4560 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:13:44.483011 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478910 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:13:44.483011 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478913 4560 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:13:44.483011 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478915 4560 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:13:44.483011 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478918 4560 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:13:44.483011 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478920 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:13:44.483011 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478923 4560 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:13:44.483011 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478925 4560 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:13:44.483011 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478928 4560 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:13:44.483011 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478930 4560 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:13:44.483011 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478933 4560 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:13:44.483513 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478935 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:13:44.483513 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478938 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:13:44.483513 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478942 4560 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:13:44.483513 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478944 4560 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:13:44.483513 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478947 4560 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:13:44.483513 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478950 4560 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:13:44.483513 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478952 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:13:44.483513 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478955 4560 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:13:44.483513 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478957 4560 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:13:44.483513 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478960 4560 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:13:44.483513 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478962 4560 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:13:44.483513 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478966 4560 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:13:44.483513 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478968 4560 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:13:44.483513 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478971 4560 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:13:44.483513 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478974 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:13:44.483513 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478977 4560 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:13:44.483513 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478980 4560 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:13:44.483513 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478982 4560 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:13:44.483513 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478985 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:13:44.483513 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478988 4560 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:13:44.483988 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478990 4560 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:13:44.483988 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478993 4560 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:13:44.483988 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.478996 4560 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:13:44.483988 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479000 4560 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:13:44.483988 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479003 4560 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:13:44.483988 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479007 4560 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:13:44.483988 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479009 4560 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:13:44.483988 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479012 4560 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:13:44.483988 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479014 4560 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:13:44.483988 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479017 4560 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:13:44.483988 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479019 4560 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:13:44.483988 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479023 4560 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:13:44.483988 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479026 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:13:44.483988 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479031 4560 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:13:44.483988 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479034 4560 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:13:44.483988 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479037 4560 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:13:44.483988 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479039 4560 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:13:44.483988 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479042 4560 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:13:44.483988 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479044 4560 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:13:44.484590 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479047 4560 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:13:44.484590 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479049 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:13:44.484590 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479052 4560 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:13:44.484590 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479054 4560 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:13:44.484590 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479056 4560 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:13:44.484590 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479059 4560 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:13:44.484590 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479062 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:13:44.484590 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479065 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:13:44.484590 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479068 4560 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:13:44.484590 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479070 4560 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:13:44.484590 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479072 4560 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:13:44.484590 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479075 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:13:44.484590 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479077 4560 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:13:44.484590 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479080 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:13:44.484590 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479082 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:13:44.484590 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479085 4560 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:13:44.484590 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479088 4560 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:13:44.484590 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479090 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:13:44.484590 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479093 4560 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:13:44.484590 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479095 4560 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:13:44.485081 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479098 4560 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:13:44.485081 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479100 4560 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:13:44.485081 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479103 4560 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:13:44.485081 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479198 4560 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:13:44.485081 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479205 4560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:13:44.485081 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479211 4560 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:13:44.485081 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479218 4560 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:13:44.485081 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479223 4560 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:13:44.485081 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479226 4560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:13:44.485081 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479231 4560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:13:44.485081 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479235 4560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:13:44.485081 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479238 4560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:13:44.485081 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479241 4560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:13:44.485081 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479245 4560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:13:44.485081 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479248 4560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:13:44.485081 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479251 4560 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:13:44.485081 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479254 4560 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:13:44.485081 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479257 4560 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:13:44.485081 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479260 4560 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:13:44.485081 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479263 4560 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:13:44.485081 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479266 4560 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:13:44.485081 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479270 4560 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:13:44.485081 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479272 4560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:13:44.485667 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479276 4560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:13:44.485667 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479279 4560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:13:44.485667 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479283 4560 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:13:44.485667 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479287 4560 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:13:44.485667 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479289 4560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:13:44.485667 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479293 4560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:13:44.485667 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479296 4560 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:13:44.485667 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479299 4560 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:13:44.485667 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479302 4560 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:13:44.485667 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479305 4560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:13:44.485667 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479308 4560 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:13:44.485667 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479311 4560 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:13:44.485667 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479315 4560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:13:44.485667 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479318 4560 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:13:44.485667 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479321 4560 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:13:44.485667 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479325 4560 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:13:44.485667 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479329 4560 flags.go:64] FLAG: --enable-server="true" Oct 30 21:13:44.485667 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479332 4560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:13:44.485667 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479349 4560 flags.go:64] FLAG: --event-burst="100" Oct 30 21:13:44.485667 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479353 4560 flags.go:64] FLAG: --event-qps="50" Oct 30 21:13:44.485667 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479356 4560 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:13:44.485667 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479359 4560 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:13:44.485667 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479362 4560 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:13:44.485667 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479366 4560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:13:44.486295 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479369 4560 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:13:44.486295 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479372 4560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:13:44.486295 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479375 4560 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:13:44.486295 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479378 4560 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:13:44.486295 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479381 4560 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:13:44.486295 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479384 4560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:13:44.486295 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479387 4560 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:13:44.486295 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479391 4560 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:13:44.486295 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479393 4560 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:13:44.486295 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479396 4560 flags.go:64] FLAG: --feature-gates="" Oct 30 21:13:44.486295 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479400 4560 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:13:44.486295 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479403 4560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:13:44.486295 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479407 4560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:13:44.486295 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479410 4560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:13:44.486295 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479413 4560 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:13:44.486295 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479416 4560 flags.go:64] FLAG: --help="false" Oct 30 21:13:44.486295 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479419 4560 flags.go:64] FLAG: --hostname-override="ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:44.486295 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479422 4560 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:13:44.486295 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479425 4560 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:13:44.486295 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479428 4560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:13:44.486295 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479431 4560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:13:44.486295 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479434 4560 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:13:44.486295 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479437 4560 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:13:44.486988 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479442 4560 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:13:44.486988 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479447 4560 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:13:44.486988 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479450 4560 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:13:44.486988 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479453 4560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:13:44.486988 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479456 4560 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:13:44.486988 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479459 4560 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:13:44.486988 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479462 4560 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:13:44.486988 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479465 4560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:13:44.486988 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479468 4560 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:13:44.486988 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479471 4560 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:13:44.486988 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479474 4560 flags.go:64] FLAG: --lock-file="" Oct 30 21:13:44.486988 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479477 4560 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:13:44.486988 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479480 4560 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:13:44.486988 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479483 4560 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:13:44.486988 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479488 4560 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:13:44.486988 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479491 4560 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:13:44.486988 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479494 4560 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:13:44.486988 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479497 4560 flags.go:64] FLAG: --logging-format="text" Oct 30 21:13:44.486988 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479500 4560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:13:44.486988 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479504 4560 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:13:44.486988 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479508 4560 flags.go:64] FLAG: --manifest-url="" Oct 30 21:13:44.486988 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479511 4560 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:13:44.486988 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479515 4560 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:13:44.486988 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479519 4560 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:13:44.486988 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479522 4560 flags.go:64] FLAG: --max-pods="110" Oct 30 21:13:44.487863 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479525 4560 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:13:44.487863 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479529 4560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:13:44.487863 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479532 4560 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:13:44.487863 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479535 4560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:13:44.487863 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479538 4560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:13:44.487863 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479540 4560 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:13:44.487863 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479543 4560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:13:44.487863 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479551 4560 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:13:44.487863 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479556 4560 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:13:44.487863 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479558 4560 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:13:44.487863 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479561 4560 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:13:44.487863 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479564 4560 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:13:44.487863 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479567 4560 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:13:44.487863 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479571 4560 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:13:44.487863 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479574 4560 flags.go:64] FLAG: --port="10250" Oct 30 21:13:44.487863 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479577 4560 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:13:44.487863 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479579 4560 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0e1ea79f4fa19c526" Oct 30 21:13:44.487863 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479583 4560 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:13:44.487863 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479586 4560 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:13:44.487863 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479589 4560 flags.go:64] FLAG: --register-node="true" Oct 30 21:13:44.487863 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479592 4560 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:13:44.487863 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479595 4560 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:13:44.487863 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479598 4560 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:13:44.487863 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479601 4560 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:13:44.487863 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479604 4560 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:13:44.487863 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479608 4560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:13:44.488951 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479611 4560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:13:44.488951 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479614 4560 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:13:44.488951 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479617 4560 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:13:44.488951 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479620 4560 flags.go:64] FLAG: --runonce="false" Oct 30 21:13:44.488951 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479623 4560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:13:44.488951 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479626 4560 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:13:44.488951 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479629 4560 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:13:44.488951 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479632 4560 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:13:44.488951 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479635 4560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:13:44.488951 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479638 4560 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:13:44.488951 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479641 4560 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:13:44.488951 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479644 4560 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:13:44.488951 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479647 4560 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:13:44.488951 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479650 4560 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:13:44.488951 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479655 4560 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:13:44.488951 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479657 4560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:13:44.488951 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479661 4560 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:13:44.488951 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479664 4560 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:13:44.488951 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479667 4560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:13:44.488951 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479672 4560 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:13:44.488951 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479675 4560 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:13:44.488951 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479678 4560 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:13:44.488951 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479682 4560 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:13:44.488951 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479685 4560 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:13:44.488951 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479688 4560 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:13:44.489670 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479691 4560 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:13:44.489670 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479694 4560 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:13:44.489670 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479697 4560 flags.go:64] FLAG: --v="2" Oct 30 21:13:44.489670 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479701 4560 flags.go:64] FLAG: --version="false" Oct 30 21:13:44.489670 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479705 4560 flags.go:64] FLAG: --vmodule="" Oct 30 21:13:44.489670 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479709 4560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:13:44.489670 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.479712 4560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:13:44.489670 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479839 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:13:44.489670 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479842 4560 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:13:44.489670 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479845 4560 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:13:44.489670 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479848 4560 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:13:44.489670 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479851 4560 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:13:44.489670 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479854 4560 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:13:44.489670 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479856 4560 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:13:44.489670 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479859 4560 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:13:44.489670 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479862 4560 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:13:44.489670 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479864 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:13:44.489670 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479867 4560 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:13:44.489670 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479869 4560 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:13:44.489670 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479872 4560 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:13:44.489670 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479875 4560 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:13:44.490354 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479879 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:13:44.490354 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479882 4560 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:13:44.490354 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479885 4560 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:13:44.490354 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479888 4560 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:13:44.490354 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479890 4560 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:13:44.490354 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479893 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:13:44.490354 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479895 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:13:44.490354 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479899 4560 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:13:44.490354 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479901 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:13:44.490354 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479904 4560 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:13:44.490354 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479907 4560 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:13:44.490354 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479909 4560 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:13:44.490354 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479912 4560 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:13:44.490354 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479914 4560 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:13:44.490354 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479917 4560 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:13:44.490354 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479920 4560 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:13:44.490354 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479922 4560 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:13:44.490354 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479925 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:13:44.490354 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479927 4560 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:13:44.490354 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479930 4560 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:13:44.491028 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479932 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:13:44.491028 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479935 4560 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:13:44.491028 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479938 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:13:44.491028 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479940 4560 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:13:44.491028 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479943 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:13:44.491028 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479945 4560 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:13:44.491028 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479948 4560 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:13:44.491028 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479950 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:13:44.491028 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479953 4560 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:13:44.491028 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479955 4560 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:13:44.491028 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479958 4560 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:13:44.491028 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479961 4560 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:13:44.491028 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479965 4560 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:13:44.491028 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479968 4560 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:13:44.491028 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479970 4560 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:13:44.491028 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479973 4560 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:13:44.491028 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479975 4560 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:13:44.491028 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479978 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:13:44.491028 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479980 4560 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:13:44.491028 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479983 4560 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:13:44.491577 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479986 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:13:44.491577 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479989 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:13:44.491577 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479991 4560 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:13:44.491577 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479994 4560 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:13:44.491577 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.479997 4560 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:13:44.491577 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480000 4560 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:13:44.491577 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480002 4560 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:13:44.491577 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480006 4560 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:13:44.491577 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480010 4560 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:13:44.491577 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480013 4560 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:13:44.491577 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480016 4560 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:13:44.491577 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480020 4560 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:13:44.491577 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480022 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:13:44.491577 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480025 4560 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:13:44.491577 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480027 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:13:44.491577 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480030 4560 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:13:44.491577 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480033 4560 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:13:44.491577 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480035 4560 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:13:44.491577 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480038 4560 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:13:44.492068 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480040 4560 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:13:44.492068 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480043 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:13:44.492068 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480045 4560 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:13:44.492068 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480048 4560 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:13:44.492068 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480051 4560 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:13:44.492068 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480062 4560 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:13:44.492068 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480065 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:13:44.492068 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480069 4560 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:13:44.492068 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480074 4560 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:13:44.492068 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480077 4560 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:13:44.492068 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480080 4560 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:13:44.492068 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480083 4560 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:13:44.492068 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480086 4560 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:13:44.492068 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480088 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:13:44.492068 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480092 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:13:44.492068 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480094 4560 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:13:44.492068 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480097 4560 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:13:44.492068 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480100 4560 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:13:44.492068 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480103 4560 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:13:44.492684 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480105 4560 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:13:44.492684 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480108 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:13:44.492684 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480110 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:13:44.492684 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480113 4560 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:13:44.492684 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480116 4560 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:13:44.492684 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480120 4560 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:13:44.492684 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480123 4560 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:13:44.492684 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480126 4560 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:13:44.492684 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480128 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:13:44.492684 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480131 4560 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:13:44.492684 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480134 4560 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:13:44.492684 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480136 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:13:44.492684 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480139 4560 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:13:44.492684 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480142 4560 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:13:44.492684 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.480167 4560 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:13:44.492684 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480200 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:13:44.493136 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480204 4560 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:13:44.493136 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480207 4560 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:13:44.493136 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480210 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:13:44.493136 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480214 4560 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:13:44.493136 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480217 4560 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:13:44.493136 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480220 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:13:44.493136 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480222 4560 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:13:44.493136 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480225 4560 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:13:44.493136 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480228 4560 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:13:44.493136 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480231 4560 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:13:44.493136 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480233 4560 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:13:44.493136 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480236 4560 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:13:44.493136 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480239 4560 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:13:44.493136 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480241 4560 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:13:44.493136 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480244 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:13:44.493136 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480247 4560 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:13:44.493136 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480249 4560 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:13:44.493136 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480252 4560 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:13:44.493136 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480254 4560 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:13:44.493680 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480257 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:13:44.493680 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480259 4560 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:13:44.493680 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480263 4560 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:13:44.493680 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480266 4560 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:13:44.493680 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480269 4560 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:13:44.493680 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480271 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:13:44.493680 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480274 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:13:44.493680 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480276 4560 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:13:44.493680 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480279 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:13:44.493680 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480281 4560 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:13:44.493680 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480284 4560 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:13:44.493680 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480287 4560 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:13:44.493680 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480289 4560 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:13:44.493680 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480292 4560 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:13:44.493680 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480295 4560 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:13:44.493680 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480297 4560 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:13:44.493680 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480301 4560 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:13:44.493680 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480304 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:13:44.493680 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480306 4560 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:13:44.493680 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480309 4560 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:13:44.494233 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480311 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:13:44.494233 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480314 4560 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:13:44.494233 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480316 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:13:44.494233 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480319 4560 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:13:44.494233 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480322 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:13:44.494233 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480324 4560 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:13:44.494233 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480327 4560 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:13:44.494233 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480330 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:13:44.494233 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480332 4560 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:13:44.494233 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480366 4560 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:13:44.494233 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480369 4560 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:13:44.494233 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480372 4560 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:13:44.494233 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480375 4560 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:13:44.494233 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480378 4560 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:13:44.494233 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480381 4560 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:13:44.494233 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480384 4560 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:13:44.494233 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480387 4560 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:13:44.494233 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480389 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:13:44.494233 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480392 4560 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:13:44.494233 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480396 4560 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:13:44.494737 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480399 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:13:44.494737 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480402 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:13:44.494737 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480405 4560 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:13:44.494737 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480407 4560 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:13:44.494737 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480410 4560 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:13:44.494737 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480413 4560 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:13:44.494737 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480415 4560 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:13:44.494737 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480419 4560 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:13:44.494737 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480426 4560 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:13:44.494737 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480429 4560 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:13:44.494737 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480432 4560 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:13:44.494737 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480435 4560 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:13:44.494737 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480437 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:13:44.494737 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480440 4560 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:13:44.494737 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480443 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:13:44.494737 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480445 4560 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:13:44.494737 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480448 4560 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:13:44.494737 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480451 4560 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:13:44.494737 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480453 4560 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:13:44.495199 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480456 4560 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:13:44.495199 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480459 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:13:44.495199 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480462 4560 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:13:44.495199 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480464 4560 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:13:44.495199 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480467 4560 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:13:44.495199 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480470 4560 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:13:44.495199 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480472 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:13:44.495199 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480478 4560 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:13:44.495199 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480481 4560 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:13:44.495199 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480484 4560 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:13:44.495199 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480487 4560 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:13:44.495199 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480489 4560 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:13:44.495199 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480492 4560 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:13:44.495199 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480495 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:13:44.495199 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480497 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:13:44.495199 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480500 4560 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:13:44.495199 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480502 4560 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:13:44.495199 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480505 4560 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:13:44.495199 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480509 4560 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:13:44.495700 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480512 4560 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:13:44.495700 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480514 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:13:44.495700 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480518 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:13:44.495700 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480521 4560 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:13:44.495700 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480524 4560 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:13:44.495700 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480526 4560 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:13:44.495700 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480529 4560 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:13:44.495700 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.480531 4560 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:13:44.495700 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.484061 4560 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:13:44.495700 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.484075 4560 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:13:44.495700 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.484092 4560 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:13:44.495700 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.484097 4560 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:13:44.495700 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484152 4560 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:13:44.495700 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484158 4560 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:13:44.495700 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484161 4560 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:13:44.495700 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484165 4560 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:13:44.495700 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484168 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:13:44.495700 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484171 4560 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:13:44.495700 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484174 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:13:44.495700 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484177 4560 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:13:44.496195 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484180 4560 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:13:44.496195 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484182 4560 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:13:44.496195 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484185 4560 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:13:44.496195 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484187 4560 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:13:44.496195 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484190 4560 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:13:44.496195 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484193 4560 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:13:44.496195 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484195 4560 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:13:44.496195 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484198 4560 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:13:44.496195 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484200 4560 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:13:44.496195 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484203 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:13:44.496195 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484206 4560 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:13:44.496195 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484208 4560 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:13:44.496195 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484210 4560 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:13:44.496195 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484213 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:13:44.496195 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484216 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:13:44.496195 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484219 4560 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:13:44.496195 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484222 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:13:44.496195 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484225 4560 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:13:44.496195 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484227 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:13:44.496195 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484230 4560 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:13:44.496893 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484233 4560 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:13:44.496893 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484236 4560 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:13:44.496893 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484239 4560 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:13:44.496893 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484242 4560 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:13:44.496893 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484244 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:13:44.496893 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484247 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:13:44.496893 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484250 4560 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:13:44.496893 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484253 4560 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:13:44.496893 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484256 4560 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:13:44.496893 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484258 4560 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:13:44.496893 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484261 4560 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:13:44.496893 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484264 4560 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:13:44.496893 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484266 4560 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:13:44.496893 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484269 4560 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:13:44.496893 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484271 4560 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:13:44.496893 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484274 4560 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:13:44.496893 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484276 4560 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:13:44.496893 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484279 4560 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:13:44.496893 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484282 4560 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:13:44.496893 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484284 4560 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:13:44.497402 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484287 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:13:44.497402 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484290 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:13:44.497402 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484292 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:13:44.497402 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484295 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:13:44.497402 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484298 4560 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:13:44.497402 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484301 4560 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:13:44.497402 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484303 4560 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:13:44.497402 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484306 4560 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:13:44.497402 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484308 4560 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:13:44.497402 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484311 4560 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:13:44.497402 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484313 4560 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:13:44.497402 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484317 4560 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:13:44.497402 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484321 4560 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:13:44.497402 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484324 4560 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:13:44.497402 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484327 4560 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:13:44.497402 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484329 4560 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:13:44.497402 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484332 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:13:44.497402 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484357 4560 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:13:44.497402 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484361 4560 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:13:44.497402 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484365 4560 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:13:44.497886 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484367 4560 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:13:44.497886 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484370 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:13:44.497886 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484373 4560 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:13:44.497886 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484376 4560 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:13:44.497886 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484379 4560 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:13:44.497886 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484381 4560 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:13:44.497886 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484384 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:13:44.497886 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484386 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:13:44.497886 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484388 4560 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:13:44.497886 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484391 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:13:44.497886 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484394 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:13:44.497886 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484396 4560 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:13:44.497886 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484399 4560 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:13:44.497886 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484401 4560 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:13:44.497886 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484404 4560 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:13:44.497886 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484407 4560 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:13:44.497886 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484411 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:13:44.497886 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484415 4560 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:13:44.497886 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484418 4560 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:13:44.498354 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484421 4560 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:13:44.498354 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484423 4560 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:13:44.498354 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484426 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:13:44.498354 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484429 4560 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:13:44.498354 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484431 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:13:44.498354 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484434 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:13:44.498354 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484437 4560 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:13:44.498354 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484440 4560 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:13:44.498354 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484442 4560 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:13:44.498354 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484445 4560 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:13:44.498354 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484448 4560 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:13:44.498354 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484450 4560 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:13:44.498354 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484453 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:13:44.498354 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484456 4560 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:13:44.498354 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484458 4560 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:13:44.498354 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484461 4560 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:13:44.498354 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484463 4560 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:13:44.498354 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484466 4560 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:13:44.498354 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484469 4560 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:13:44.498831 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.484496 4560 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:13:44.498831 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484628 4560 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:13:44.498831 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484633 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:13:44.498831 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484636 4560 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:13:44.498831 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484639 4560 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:13:44.498831 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484642 4560 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:13:44.498831 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484644 4560 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:13:44.498831 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484648 4560 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:13:44.498831 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484651 4560 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:13:44.498831 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484654 4560 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:13:44.498831 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484657 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:13:44.498831 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484660 4560 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:13:44.498831 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484663 4560 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:13:44.498831 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484666 4560 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:13:44.498831 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484668 4560 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:13:44.498831 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484671 4560 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:13:44.499217 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484674 4560 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:13:44.499217 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484676 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:13:44.499217 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484679 4560 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:13:44.499217 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484682 4560 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:13:44.499217 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484684 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:13:44.499217 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484687 4560 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:13:44.499217 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484689 4560 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:13:44.499217 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484691 4560 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:13:44.499217 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484694 4560 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:13:44.499217 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484696 4560 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:13:44.499217 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484699 4560 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:13:44.499217 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484702 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:13:44.499217 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484704 4560 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:13:44.499217 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484707 4560 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:13:44.499217 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484709 4560 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:13:44.499217 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484712 4560 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:13:44.499217 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484715 4560 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:13:44.499217 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484717 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:13:44.499217 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484719 4560 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:13:44.499217 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484722 4560 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:13:44.499715 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484724 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:13:44.499715 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484727 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:13:44.499715 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484729 4560 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:13:44.499715 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484732 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:13:44.499715 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484734 4560 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:13:44.499715 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484736 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:13:44.499715 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484739 4560 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:13:44.499715 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484741 4560 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:13:44.499715 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484744 4560 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:13:44.499715 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484746 4560 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:13:44.499715 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484749 4560 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:13:44.499715 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484751 4560 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:13:44.499715 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484754 4560 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:13:44.499715 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484756 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:13:44.499715 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484759 4560 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:13:44.499715 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484762 4560 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:13:44.499715 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484764 4560 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:13:44.499715 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484767 4560 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:13:44.499715 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484769 4560 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:13:44.500217 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484772 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:13:44.500217 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484775 4560 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:13:44.500217 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484777 4560 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:13:44.500217 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484780 4560 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:13:44.500217 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484782 4560 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:13:44.500217 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484786 4560 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:13:44.500217 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484788 4560 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:13:44.500217 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484791 4560 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:13:44.500217 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484793 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:13:44.500217 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484796 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:13:44.500217 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484799 4560 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:13:44.500217 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484817 4560 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:13:44.500217 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484821 4560 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:13:44.500217 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484824 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:13:44.500217 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484827 4560 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:13:44.500217 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484829 4560 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:13:44.500217 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484832 4560 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:13:44.500217 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484834 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:13:44.500217 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484837 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:13:44.500680 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484840 4560 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:13:44.500680 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484842 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:13:44.500680 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484847 4560 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:13:44.500680 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484850 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:13:44.500680 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484852 4560 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:13:44.500680 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484855 4560 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:13:44.500680 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484858 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:13:44.500680 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484860 4560 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:13:44.500680 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484863 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:13:44.500680 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484867 4560 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:13:44.500680 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484870 4560 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:13:44.500680 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484873 4560 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:13:44.500680 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484875 4560 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:13:44.500680 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484878 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:13:44.500680 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484881 4560 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:13:44.500680 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484884 4560 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:13:44.500680 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484886 4560 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:13:44.500680 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484889 4560 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:13:44.500680 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484891 4560 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:13:44.501131 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484895 4560 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:13:44.501131 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484898 4560 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:13:44.501131 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484900 4560 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:13:44.501131 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484903 4560 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:13:44.501131 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484906 4560 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:13:44.501131 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484908 4560 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:13:44.501131 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484911 4560 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:13:44.501131 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484915 4560 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:13:44.501131 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484918 4560 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:13:44.501131 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484922 4560 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:13:44.501131 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484925 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:13:44.501131 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484927 4560 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:13:44.501131 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484930 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:13:44.501131 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484933 4560 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:13:44.501131 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.484955 4560 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:13:44.501131 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484984 4560 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:13:44.501632 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484989 4560 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:13:44.501632 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484993 4560 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:13:44.501632 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.484997 4560 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:13:44.501632 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485000 4560 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:13:44.501632 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485003 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:13:44.501632 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485006 4560 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:13:44.501632 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485010 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:13:44.501632 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485013 4560 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:13:44.501632 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485015 4560 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:13:44.501632 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485018 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:13:44.501632 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485021 4560 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:13:44.501632 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485024 4560 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:13:44.501632 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485027 4560 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:13:44.501632 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485029 4560 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:13:44.501632 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485032 4560 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:13:44.501632 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485034 4560 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:13:44.501632 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485037 4560 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:13:44.501632 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485041 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:13:44.501632 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485043 4560 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:13:44.502083 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485046 4560 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:13:44.502083 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485048 4560 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:13:44.502083 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485051 4560 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:13:44.502083 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485053 4560 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:13:44.502083 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485056 4560 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:13:44.502083 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485058 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:13:44.502083 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485061 4560 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:13:44.502083 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485063 4560 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:13:44.502083 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485066 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:13:44.502083 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485069 4560 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:13:44.502083 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485071 4560 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:13:44.502083 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485074 4560 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:13:44.502083 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485076 4560 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:13:44.502083 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485082 4560 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:13:44.502083 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485085 4560 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:13:44.502083 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485088 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:13:44.502083 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485091 4560 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:13:44.502083 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485094 4560 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:13:44.502083 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485097 4560 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:13:44.502083 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485099 4560 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:13:44.502688 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485102 4560 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:13:44.502688 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485105 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:13:44.502688 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485107 4560 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:13:44.502688 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485110 4560 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:13:44.502688 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485112 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:13:44.502688 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485115 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:13:44.502688 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485117 4560 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:13:44.502688 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485120 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:13:44.502688 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485123 4560 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:13:44.502688 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485125 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:13:44.502688 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485128 4560 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:13:44.502688 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485131 4560 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:13:44.502688 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485133 4560 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:13:44.502688 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485136 4560 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:13:44.502688 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485138 4560 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:13:44.502688 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485141 4560 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:13:44.502688 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485143 4560 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:13:44.502688 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485146 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:13:44.502688 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485148 4560 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:13:44.503325 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485152 4560 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:13:44.503325 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485154 4560 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:13:44.503325 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485157 4560 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:13:44.503325 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485159 4560 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:13:44.503325 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485162 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:13:44.503325 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485165 4560 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:13:44.503325 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485170 4560 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:13:44.503325 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485172 4560 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:13:44.503325 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485175 4560 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:13:44.503325 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485177 4560 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:13:44.503325 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485180 4560 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:13:44.503325 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485182 4560 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:13:44.503325 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485185 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:13:44.503325 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485188 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:13:44.503325 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485191 4560 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:13:44.503325 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485193 4560 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:13:44.503325 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485196 4560 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:13:44.503325 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485198 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:13:44.503325 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485201 4560 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:13:44.503325 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485203 4560 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:13:44.504072 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485206 4560 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:13:44.504072 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485208 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:13:44.504072 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485210 4560 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:13:44.504072 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485213 4560 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:13:44.504072 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485215 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:13:44.504072 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485218 4560 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:13:44.504072 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485221 4560 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:13:44.504072 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485223 4560 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:13:44.504072 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485226 4560 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:13:44.504072 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485228 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:13:44.504072 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485231 4560 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:13:44.504072 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485233 4560 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:13:44.504072 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485237 4560 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:13:44.504072 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485241 4560 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:13:44.504072 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485244 4560 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:13:44.504072 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485247 4560 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:13:44.504072 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485251 4560 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:13:44.504072 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485254 4560 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:13:44.504855 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485258 4560 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:13:44.504855 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485261 4560 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:13:44.504855 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485264 4560 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:13:44.504855 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485266 4560 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:13:44.504855 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485269 4560 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:13:44.504855 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485271 4560 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:13:44.504855 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485275 4560 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:13:44.504855 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485277 4560 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:13:44.504855 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:44.485280 4560 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:13:44.504855 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.485326 4560 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:13:44.504855 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.486530 4560 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 21:13:44.504855 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.486583 4560 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 21:13:44.504855 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.486776 4560 server.go:1014] "Starting client certificate rotation" Oct 30 21:13:44.504855 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.486920 4560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:13:44.504855 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.486965 4560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:08:05 +0000 UTC" deadline="2028-04-18 00:45:44.522094492 +0000 UTC" Oct 30 21:13:44.504855 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.486999 4560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12843h32m0.03509742s" Oct 30 21:13:44.504855 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.487219 4560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:13:44.505499 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.489645 4560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:13:44.505499 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.491101 4560 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:13:44.505499 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.492664 4560 log.go:25] "Validated CRI v1 image API" Oct 30 21:13:44.505499 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.493412 4560 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:13:44.505499 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.494917 4560 fs.go:135] Filesystem UUIDs: map[4a4bd9b2-9de1-493b-b9de-fa09a6e7f48f:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d02f50c3-c68e-4a70-acb7-cdc6206c5992:/dev/nvme0n1p4] Oct 30 21:13:44.505643 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.494936 4560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0bc90de4c74ac62133ee2f84e8d2364207b45a0853fc7b80529262f8213ce9c2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0bc90de4c74ac62133ee2f84e8d2364207b45a0853fc7b80529262f8213ce9c2/userdata/shm major:0 minor:64 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/11f55cd81c2b0cd75bad91a53655e3dcbf2481f2269d44de8a43f7d6489a5d66/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/11f55cd81c2b0cd75bad91a53655e3dcbf2481f2269d44de8a43f7d6489a5d66/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/15d280b6927840bc8d5e670ceaf0d1ad63c76214be7bdfb29b41789dfb1e849b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/15d280b6927840bc8d5e670ceaf0d1ad63c76214be7bdfb29b41789dfb1e849b/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/390843099a133d9ae571a91d7988a0eb49448d83668b3a037d62a0de90d0396f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/390843099a133d9ae571a91d7988a0eb49448d83668b3a037d62a0de90d0396f/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5b0979854386d0a46f5c2c6c58f661c5f84538da48cee36194af915536819a42/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5b0979854386d0a46f5c2c6c58f661c5f84538da48cee36194af915536819a42/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6d6f28503f3ba781641478ee756d05fee509b2b0f70a1825e79bbb2576ee9849/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6d6f28503f3ba781641478ee756d05fee509b2b0f70a1825e79bbb2576ee9849/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/81378ec2ad2c76d0f97979360af798bf2d824bde7db7842663a8373b224f5bb0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/81378ec2ad2c76d0f97979360af798bf2d824bde7db7842663a8373b224f5bb0/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/81ee5ef9d70c948a11cb050642ec76add87d6871d46aa1445bd0a65c35fc9d5d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/81ee5ef9d70c948a11cb050642ec76add87d6871d46aa1445bd0a65c35fc9d5d/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/821d2084423f5098c7e92e6ab480a1dae74e29ea70ce538ce25533941e5ec63b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/821d2084423f5098c7e92e6ab480a1dae74e29ea70ce538ce25533941e5ec63b/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8312a77f41c875aae9eea8ead18164c2574c4b690333c9efaae39ac09e4f37e0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8312a77f41c875aae9eea8ead18164c2574c4b690333c9efaae39ac09e4f37e0/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9a77293a9478bf036b3af7ffc7cb8489a1d116738bb93f46336e37ed1c9b7506/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9a77293a9478bf036b3af7ffc7cb8489a1d116738bb93f46336e37ed1c9b7506/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c2e5dac0e9ba9d2162d1da40dd6bd04b66a2a324a2fd34b57dedc7cb1bcdc7ad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c2e5dac0e9ba9d2162d1da40dd6bd04b66a2a324a2fd34b57dedc7cb1bcdc7ad/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c580ec59787cb3cc0e10e37be16b38aa039208e54b726780cea4cf2f5d570fb2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c580ec59787cb3cc0e10e37be16b38aa039208e54b726780cea4cf2f5d570fb2/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/caacc81147f4b137a7f1f62412f4b237d9c264e996823f9897bf2da9577056c3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/caacc81147f4b137a7f1f62412f4b237d9c264e996823f9897bf2da9577056c3/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d1d8eebf965a999029a65e81b2c59c8511529ef74c8e1ff1c67290be1b4fdb05/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d1d8eebf965a999029a65e81b2c59c8511529ef74c8e1ff1c67290be1b4fdb05/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dd42db4be3a7cb80a93a5bf0a381856b58cae9e07d84532c87a1eab458a25b76/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dd42db4be3a7cb80a93a5bf0a381856b58cae9e07d84532c87a1eab458a25b76/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f929d541705328cc0720bdb06c631f4ff12e9e449eebb887da4e855b09568ec9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f929d541705328cc0720bdb06c631f4ff12e9e449eebb887da4e855b09568ec9/userdata/shm major:0 minor:264 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05d96464-f946-4a83-88e5-ececfee30b4c/volumes/kubernetes.io~projected/kube-api-access-cctxz:{mountpoint:/var/lib/kubelet/pods/05d96464-f946-4a83-88e5-ececfee30b4c/volumes/kubernetes.io~projected/kube-api-access-cctxz major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08f1ac32-f49e-4aa2-9ab8-8a1aac7bef36/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/08f1ac32-f49e-4aa2-9ab8-8a1aac7bef36/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17696d35-744c-434c-94de-33ec719feac9/volumes/kubernetes.io~projected/kube-api-access-5ghv7:{mountpoint:/var/lib/kubelet/pods/17696d35-744c-434c-94de-33ec719feac9/volumes/kubernetes.io~projected/kube-api-access-5ghv7 major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17696d35-744c-434c-94de-33ec719feac9/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/17696d35-744c-434c-94de-33ec719feac9/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18d4542b-53c1-473f-9605-c82fd5a642f4/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/18d4542b-53c1-473f-9605-c82fd5a642f4/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18d4542b-53c1-473f-9605-c82fd5a642f4/volumes/kubernetes.io~projected/kube-api-access-k52xf:{mountpoint:/var/lib/kubelet/pods/18d4542b-53c1-473f-9605-c82fd5a642f4/volumes/kubernetes.io~projected/kube-api-access-k52xf major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18d4542b-53c1-473f-9605-c82fd5a642f4/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/18d4542b-53c1-473f-9605-c82fd5a642f4/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19538cda-3ba8-4bad-bb47-23a2511736e7/volumes/kubernetes.io~projected/kube-api-access-bsrnm:{mountpoint:/var/lib/kubelet/pods/19538cda-3ba8-4bad-bb47-23a2511736e7/volumes/kubernetes.io~projected/kube-api-access-bsrnm major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d4aae53-afc9-4373-953d-b9b7c323e3ee/volumes/kubernetes.io~projected/kube-api-access-tvgz2:{mountpoint:/var/lib/kubelet/pods/1d4aae53-afc9-4373-953d-b9b7c323e3ee/volumes/kubernetes.io~projected/kube-api-access-tvgz2 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25ca3b5e-7eba-4411-be09-5da0fe41e473/volumes/kubernetes.io~projected/kube-api-access-mvbqm:{mountpoint:/var/lib/kubelet/pods/25ca3b5e-7eba-4411-be09-5da0fe41e473/volumes/kubernetes.io~projected/kube-api-access-mvbqm major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c877970-90d9-43ba-9c9b-4a22fbda1dc0/volumes/kubernetes.io~projected/kube-api-access-k8snd:{mountpoint:/var/lib/kubelet/pods/2c877970-90d9-43ba-9c9b-4a22fbda1dc0/volumes/kubernetes.io~projected/kube-api-access-k8snd major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30bde564-b785-4359-9377-ffec880fb18c/volumes/kubernetes.io~projected/kube-api-access-l8884:{mountpoint:/var/lib/kubelet/pods/30bde564-b785-4359-9377-ffec880fb18c/volumes/kubernetes.io~projected/kube-api-access-l8884 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3341bf47-8c8f-4073-8758-122e04e0b138/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/3341bf47-8c8f-4073-8758-122e04e0b138/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d528ee4-1b51-4fdb-9166-56f14060d6af/volumes/kubernetes.io~projected/kube-api-access-kbrm8:{mountpoint:/var/lib/kubelet/pods/3d528ee4-1b51-4fdb-9166-56f14060d6af/volumes/kubernetes.io~projected/kube-api-access-kbrm8 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d528ee4-1b51-4fdb-9166-56f14060d6af/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/3d528ee4-1b51-4fdb-9166-56f14060d6af/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4eed118d-1567-4be4-8c90-5dc0e7478827/volumes/kubernetes.io~projected/kube-api-access-tk4wl:{mountpoint:/var/lib/kubelet/pods/4eed118d-1567-4be4-8c90-5dc0e7478827/volumes/kubernetes.io~projected/kube-api-access-tk4wl major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54b8c0a5-e1a7-4d02-a402-b3cadc139e28/volumes/kubernetes.io~projected/kube-api-access-jg66z:{mountpoint:/var/lib/kubelet/pods/54b8c0a5-e1a7-4d02-a402-b3cadc139e28/volumes/kubernetes.io~projected/kube-api-access-jg66z major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70c3c6bc-c455-4fe6-9099-1ceb92abeaf0/volumes/kubernetes.io~projected/kube-api-access-gfs8x:{mountpoint:/var/lib/kubelet/pods/70c3c6bc-c455-4fe6-9099-1ceb92abeaf0/volumes/kubernetes.io~projected/kube-api-access-gfs8x major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9597db44-1ca0-4fa8-ade7-cb55eeca9593/volumes/kubernetes.io~projected/kube-api-access-wtvrj:{mountpoint:/var/lib/kubelet/pods/9597db44-1ca0-4fa8-ade7-cb55eeca9593/volumes/kubernetes.io~projected/kube-api-access-wtvrj major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8/volumes/kubernetes.io~empty-dir/tmp major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8/volumes/kubernetes.io~projected/kube-api-access-hl9th:{mountpoint:/var/lib/kubelet/pods/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8/volumes/kubernetes.io~projected/kube-api-access-hl9th major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f285a424-0cc2-41b8-99b9-92db326ea01d/volumes/kubernetes.io~projected/kube-api-access-gl5cb:{mountpoint:/var/lib/kubelet/pods/f285a424-0cc2-41b8-99b9-92db326ea01d/volumes/kubernetes.io~projected/kube-api-access-gl5cb major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f96e4d00-99fb-4196-a1a9-83599bfd875f/volumes/kubernetes.io~projected/kube-api-access-kdp4m:{mountpoint:/var/lib/kubelet/pods/f96e4d00-99fb-4196-a1a9-83599bfd875f/volumes/kubernetes.io~projected/kube-api-access-kdp4m major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f96e4d00-99fb-4196-a1a9-83599bfd875f/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/f96e4d00-99fb-4196-a1a9-83599bfd875f/volumes/kubernetes.io~secret/signing-key major:0 minor:213 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/16ac6ffd83b0fef5d3ad57b5e367991f2490d6a7ffd82691919b529598f36352/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/8cc962eef8ca580d87244c48266776e2703791212f8b565e691be47636f9e660/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/5ee692e0d5fdb4af705d4ada7b976c5ba9e1bcf9294aa535462ac21faeca7a26/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/63c5519ef33e80add09fc597366c4b819eeda2edc106fdfdeb151538dd4d3323/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/b231a02fde0b329e1792e8e96d7f3649f6163885fd9c7bc8a6fa9b2ee82a02ed/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/25c9a7908159de4cc721e2e3510f4be9b621b0863b72ac1059eaaa67e7e70987/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/59e763930dbd2c97b448e0767184900dacf09e9e85c54df4e5e363fe71102f32/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/a8456dc0b39caeee175b1b7445b7cf6e9c4f81234cf61c2733a1504b65d54722/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/b7fe18ea09c7dd299b176b3ca9b8f779a34f3526a68b232886c2d648f006b6b4/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/0e96a4b9b41aad90af04a353edddf718ee2cc63eca6c6e492063f79bf141d55c/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/4955c4eab110e25db9a5c210dac90f516d29d605ea9bc3c8b5be7ecd9d03d198/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/174d5764e371ea2a390799aad752ab93bc415419a32be50914c7b3f48bfe7241/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/e85234d16464f91a94bc277893387c02ed8d12563aecb9a207836548be417652/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/17516d9c17f3b366f869f2cd97bd9929775d4af7db69c1b111469d6b43bd412f/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/d058e50800f382bafb4d0bf70ed6caba702b019499ebf3bba1f20853bc063ad8/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/1f1e66b97b4b14d732483e2bd8111871820a0eb544d85bd17754e47b962383fe/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/dfdc6e27c00900a33e7b3565b19eebb609672f10300f5f94113bdbbf44b50e78/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/575f5dc8ceec9d26ca674c8d08cd8b74f1ab1f6de012a6d496d459f0a23e7acd/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/e95deda5b233d5cc50c78f0f0f523368f26224ed39ec45529e2759b037322d9b/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/633b33f572bb389c34b80f4cd3be4e4e9c9f75288d4009400a9c5f2f1ff4ad0f/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/42f076a2b01b175146cee23064dd76e6d2527651bc28dfd5ffd65574c1b7e20e/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/a946335e3f69341e0a522e1e9fa1c0b6f9b24eec0b3f59af46c61abe85d18f63/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-210:{mountpoint:/var/lib/containers/storage/overlay/1bcf6f3d304d8a225e047cb90830e5862e62cad34b7cafeafb46567608c121a1/merged major:0 minor:210 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/10c218c576909f0465eeab9e78b6bf2faa915f31ad67ed2a29f2a89fb09f4156/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/497093cb3089c2e610cd8d12e2e82e5e8ee458bb3e48e717fe779b649207df4c/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/59466a48b4977255ef79322e155b7495650b73ca3f870faf570b9736a6d33f7a/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/8288bcb00f0c46d68e88cee00fd168e17ba348cead7627e60c027a5878fb5afd/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/f0e51210233655896ec88a3646ff0879295a372d4881a4196e47a4f3e4c58065/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/a96714c0c22c0acbf245afa2977b44424ea3b33d1a49e41ac358520c9d59d4b5/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/08ddc214fbd3f82d9f27654317e011d9f92ddb165b63e96b0952fe28d440e79b/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/f3b82d6bd8a098359b2d419a1140c28c677a79984591f1477626c0267adce8b1/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/6f2cd8320f7f3fc908a7483aff9775673ec86f338a121c5c04be535124deca47/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/ef60ad6294ca1aca154b45f8673ffa32f3d597f5e084df0130ea93cd8864132a/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/3f18372dee91273d0c4c43752b4e1812c430011d6633413edf8c86eb06a73bc1/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/649281ad7fa6cc324d54f793584f1b6b3f7724d5b685ef8c5e323a7de83b0501/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-62:{mountpoint:/var/lib/containers/storage/overlay/3b43070873fce0e1a3c1d8087e4fd5113c9bdfdbc901e4230fdd2825f52d420f/merged major:0 minor:62 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/460dd0b56bf32fdaecb177b8c466ccfaa46b44f74b066bfb38de33922520f7c4/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/cd8f567bd24dcc410a548a0f7db449cd5bfe77799b3e17b03435ef2dcf8d271b/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/9216c18791bb5b9e1a639cce836ce615b7758069983605e1b65fde1efc863b68/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/e7bacac45b9ed3a7b002afdedb7514e6b5691b6738884b3210b49113806710b8/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/2cb1ac5b085d73e87a6965ae5b7f31d3048cc0b5e2554b66a5058649caa917b9/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/171eb4b8e5a75645cb61589205e7825ea30484ad2fce82582f89abb29e68fa68/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/954b337f75ed8fa5be79125379e11470385b564b1468e896f8843d0d2d7c4bcc/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/b7a7d971e80e03182e5318511ded8bb2fd9b873718f6bb86e688abae010b0434/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/2c0438b1f5e1dd40527f114adbb4ee24635a7979b7cbfe6631deccdb1427673b/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 21:13:44.505805 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.501112 4560 manager.go:217] Machine: {Timestamp:2026-10-30 21:13:44.500676476 +0000 UTC m=+0.055376243 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096233 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec291bed37d7e4f77b7ff30252ec9ef6 SystemUUID:ec291bed-37d7-e4f7-7b7f-f30252ec9ef6 BootID:7d0ed3b0-8d3a-4a2d-b3aa-bffd15478260 Filesystems:[{Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f96e4d00-99fb-4196-a1a9-83599bfd875f/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6d6f28503f3ba781641478ee756d05fee509b2b0f70a1825e79bbb2576ee9849/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c2e5dac0e9ba9d2162d1da40dd6bd04b66a2a324a2fd34b57dedc7cb1bcdc7ad/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9597db44-1ca0-4fa8-ade7-cb55eeca9593/volumes/kubernetes.io~projected/kube-api-access-wtvrj DeviceMajor:0 DeviceMinor:52 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/15d280b6927840bc8d5e670ceaf0d1ad63c76214be7bdfb29b41789dfb1e849b/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8/volumes/kubernetes.io~projected/kube-api-access-hl9th DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/81378ec2ad2c76d0f97979360af798bf2d824bde7db7842663a8373b224f5bb0/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/81ee5ef9d70c948a11cb050642ec76add87d6871d46aa1445bd0a65c35fc9d5d/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/17696d35-744c-434c-94de-33ec719feac9/volumes/kubernetes.io~projected/kube-api-access-5ghv7 DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8312a77f41c875aae9eea8ead18164c2574c4b690333c9efaae39ac09e4f37e0/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/18d4542b-53c1-473f-9605-c82fd5a642f4/volumes/kubernetes.io~projected/kube-api-access-k52xf DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9a77293a9478bf036b3af7ffc7cb8489a1d116738bb93f46336e37ed1c9b7506/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/390843099a133d9ae571a91d7988a0eb49448d83668b3a037d62a0de90d0396f/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3d528ee4-1b51-4fdb-9166-56f14060d6af/volumes/kubernetes.io~projected/kube-api-access-kbrm8 DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:48 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/11f55cd81c2b0cd75bad91a53655e3dcbf2481f2269d44de8a43f7d6489a5d66/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-210 DeviceMajor:0 DeviceMinor:210 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/821d2084423f5098c7e92e6ab480a1dae74e29ea70ce538ce25533941e5ec63b/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c580ec59787cb3cc0e10e37be16b38aa039208e54b726780cea4cf2f5d570fb2/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f285a424-0cc2-41b8-99b9-92db326ea01d/volumes/kubernetes.io~projected/kube-api-access-gl5cb DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/08f1ac32-f49e-4aa2-9ab8-8a1aac7bef36/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dd42db4be3a7cb80a93a5bf0a381856b58cae9e07d84532c87a1eab458a25b76/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/caacc81147f4b137a7f1f62412f4b237d9c264e996823f9897bf2da9577056c3/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1d4aae53-afc9-4373-953d-b9b7c323e3ee/volumes/kubernetes.io~projected/kube-api-access-tvgz2 DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0bc90de4c74ac62133ee2f84e8d2364207b45a0853fc7b80529262f8213ce9c2/userdata/shm DeviceMajor:0 DeviceMinor:64 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5b0979854386d0a46f5c2c6c58f661c5f84538da48cee36194af915536819a42/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-62 DeviceMajor:0 DeviceMinor:62 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f96e4d00-99fb-4196-a1a9-83599bfd875f/volumes/kubernetes.io~projected/kube-api-access-kdp4m DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/19538cda-3ba8-4bad-bb47-23a2511736e7/volumes/kubernetes.io~projected/kube-api-access-bsrnm DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/05d96464-f946-4a83-88e5-ececfee30b4c/volumes/kubernetes.io~projected/kube-api-access-cctxz DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d1d8eebf965a999029a65e81b2c59c8511529ef74c8e1ff1c67290be1b4fdb05/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3d528ee4-1b51-4fdb-9166-56f14060d6af/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/70c3c6bc-c455-4fe6-9099-1ceb92abeaf0/volumes/kubernetes.io~projected/kube-api-access-gfs8x DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/17696d35-744c-434c-94de-33ec719feac9/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:259 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/18d4542b-53c1-473f-9605-c82fd5a642f4/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/25ca3b5e-7eba-4411-be09-5da0fe41e473/volumes/kubernetes.io~projected/kube-api-access-mvbqm DeviceMajor:0 DeviceMinor:53 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/54b8c0a5-e1a7-4d02-a402-b3cadc139e28/volumes/kubernetes.io~projected/kube-api-access-jg66z DeviceMajor:0 DeviceMinor:54 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/18d4542b-53c1-473f-9605-c82fd5a642f4/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4eed118d-1567-4be4-8c90-5dc0e7478827/volumes/kubernetes.io~projected/kube-api-access-tk4wl DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2c877970-90d9-43ba-9c9b-4a22fbda1dc0/volumes/kubernetes.io~projected/kube-api-access-k8snd DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3341bf47-8c8f-4073-8758-122e04e0b138/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/30bde564-b785-4359-9377-ffec880fb18c/volumes/kubernetes.io~projected/kube-api-access-l8884 DeviceMajor:0 DeviceMinor:51 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f929d541705328cc0720bdb06c631f4ff12e9e449eebb887da4e855b09568ec9/userdata/shm DeviceMajor:0 DeviceMinor:264 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:5b0979854386d0a MacAddress:76:b5:93:d4:97:db Speed:10000 Mtu:8901} {Name:6d6f28503f3ba78 MacAddress:6e:68:29:7d:58:d2 Speed:10000 Mtu:8901} {Name:81ee5ef9d70c948 MacAddress:e6:b9:2e:a7:4f:d0 Speed:10000 Mtu:8901} {Name:821d2084423f509 MacAddress:4a:7f:82:b4:b8:4a Speed:10000 Mtu:8901} {Name:8312a77f41c875a MacAddress:3a:ca:30:24:de:44 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:d5:e2:59:bb Speed:0 Mtu:9001} {Name:br-int MacAddress:6a:4e:38:fe:d2:87 Speed:0 Mtu:8901} {Name:c2e5dac0e9ba9d2 MacAddress:ca:47:9d:40:13:b4 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:d5:e2:59:bb Speed:0 Mtu:9001} {Name:f929d541705328c MacAddress:ea:00:30:16:bc:a0 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:ce:92:29:11:10:4f Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:fa:24:2f:c0:3c:2c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:13:44.505805 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.501373 4560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:13:44.505805 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.501415 4560 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:13:44.505805 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.501673 4560 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:13:44.505805 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.501694 4560 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-83.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:13:44.505805 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.501811 4560 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:13:44.505805 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.501819 4560 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:13:44.505805 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.501829 4560 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:13:44.505805 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.501841 4560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:13:44.505805 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.501920 4560 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:13:44.505805 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.502084 4560 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:13:44.505805 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.502176 4560 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:13:44.505805 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.502208 4560 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:13:44.505805 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.502221 4560 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:13:44.505805 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.502231 4560 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:13:44.505805 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.502239 4560 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:13:44.505805 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.502254 4560 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:13:44.505805 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.503237 4560 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:13:44.505805 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.503247 4560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:13:44.505805 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.503857 4560 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:13:44.513907 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.513892 4560 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:13:44.513907 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.513906 4560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:13:44.514000 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.513992 4560 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 21:13:44.514169 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.514158 4560 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:13:44.514209 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.514187 4560 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:13:44.514283 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.514273 4560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:13:44.514321 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.514288 4560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:13:44.514321 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.514294 4560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:13:44.514321 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.514299 4560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:13:44.514321 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.514305 4560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:13:44.514321 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.514310 4560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:13:44.514321 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.514316 4560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:13:44.514321 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.514321 4560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:13:44.514593 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.514329 4560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:13:44.514593 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.514352 4560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:13:44.514593 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.514363 4560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:13:44.514593 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.514373 4560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:13:44.514593 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.514398 4560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:13:44.514593 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.514403 4560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:13:44.517609 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.517595 4560 server.go:1263] "Started kubelet" Oct 30 21:13:44.518175 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.517664 4560 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:13:44.518175 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.517748 4560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:13:44.518175 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.517825 4560 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:13:44.518175 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.518121 4560 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:13:44.518104 ip-10-0-129-83 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:13:44.519087 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.519069 4560 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:13:44.521288 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.520556 4560 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:13:44.521288 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.520576 4560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:13:44.521288 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.520620 4560 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:13:44.521288 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.520970 4560 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:13:44.521288 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.520631 4560 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:13:44.521288 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.521019 4560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:08:06 +0000 UTC" deadline="2028-04-17 16:37:22.627198493 +0000 UTC" Oct 30 21:13:44.521288 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:13:44.520631 4560 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-83.us-west-2.compute.internal\" not found" Oct 30 21:13:44.521288 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.521051 4560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12835h23m38.106150961s" Oct 30 21:13:44.521288 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.521218 4560 factory.go:55] Registering systemd factory Oct 30 21:13:44.521288 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.521230 4560 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.523239 4560 factory.go:153] Registering CRI-O factory Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.523261 4560 factory.go:223] Registration of the crio container factory successfully Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.523322 4560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.523358 4560 factory.go:103] Registering Raw factory Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.523374 4560 manager.go:1199] Started watching for new ooms in manager Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.523524 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4eed118d-1567-4be4-8c90-5dc0e7478827" volumeName="kubernetes.io/configmap/4eed118d-1567-4be4-8c90-5dc0e7478827-config-volume" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.523570 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25ca3b5e-7eba-4411-be09-5da0fe41e473" volumeName="kubernetes.io/configmap/25ca3b5e-7eba-4411-be09-5da0fe41e473-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.523586 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25ca3b5e-7eba-4411-be09-5da0fe41e473" volumeName="kubernetes.io/projected/25ca3b5e-7eba-4411-be09-5da0fe41e473-kube-api-access-mvbqm" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.523607 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c877970-90d9-43ba-9c9b-4a22fbda1dc0" volumeName="kubernetes.io/configmap/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-multus-daemon-config" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.523694 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4eed118d-1567-4be4-8c90-5dc0e7478827" volumeName="kubernetes.io/empty-dir/4eed118d-1567-4be4-8c90-5dc0e7478827-tmp-dir" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.523708 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4eed118d-1567-4be4-8c90-5dc0e7478827" volumeName="kubernetes.io/projected/4eed118d-1567-4be4-8c90-5dc0e7478827-kube-api-access-tk4wl" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.523725 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54b8c0a5-e1a7-4d02-a402-b3cadc139e28" volumeName="kubernetes.io/projected/54b8c0a5-e1a7-4d02-a402-b3cadc139e28-kube-api-access-jg66z" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.523775 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f96e4d00-99fb-4196-a1a9-83599bfd875f" volumeName="kubernetes.io/projected/f96e4d00-99fb-4196-a1a9-83599bfd875f-kube-api-access-kdp4m" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.523798 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17696d35-744c-434c-94de-33ec719feac9" volumeName="kubernetes.io/configmap/17696d35-744c-434c-94de-33ec719feac9-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.523812 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18d4542b-53c1-473f-9605-c82fd5a642f4" volumeName="kubernetes.io/configmap/18d4542b-53c1-473f-9605-c82fd5a642f4-ovnkube-script-lib" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.523832 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25ca3b5e-7eba-4411-be09-5da0fe41e473" volumeName="kubernetes.io/configmap/25ca3b5e-7eba-4411-be09-5da0fe41e473-cni-binary-copy" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.523848 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30bde564-b785-4359-9377-ffec880fb18c" volumeName="kubernetes.io/projected/30bde564-b785-4359-9377-ffec880fb18c-kube-api-access-l8884" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.523866 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d528ee4-1b51-4fdb-9166-56f14060d6af" volumeName="kubernetes.io/secret/3d528ee4-1b51-4fdb-9166-56f14060d6af-cert" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.523900 4560 manager.go:319] Starting recovery of all containers Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.523960 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54b8c0a5-e1a7-4d02-a402-b3cadc139e28" volumeName="kubernetes.io/configmap/54b8c0a5-e1a7-4d02-a402-b3cadc139e28-serviceca" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.523980 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f96e4d00-99fb-4196-a1a9-83599bfd875f" volumeName="kubernetes.io/configmap/f96e4d00-99fb-4196-a1a9-83599bfd875f-signing-cabundle" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.524016 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f96e4d00-99fb-4196-a1a9-83599bfd875f" volumeName="kubernetes.io/secret/f96e4d00-99fb-4196-a1a9-83599bfd875f-signing-key" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.524088 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05d96464-f946-4a83-88e5-ececfee30b4c" volumeName="kubernetes.io/projected/05d96464-f946-4a83-88e5-ececfee30b4c-kube-api-access-cctxz" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.524108 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9597db44-1ca0-4fa8-ade7-cb55eeca9593" volumeName="kubernetes.io/projected/9597db44-1ca0-4fa8-ade7-cb55eeca9593-kube-api-access-wtvrj" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.524122 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f285a424-0cc2-41b8-99b9-92db326ea01d" volumeName="kubernetes.io/configmap/f285a424-0cc2-41b8-99b9-92db326ea01d-service-ca-bundle" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.524137 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08f1ac32-f49e-4aa2-9ab8-8a1aac7bef36" volumeName="kubernetes.io/secret/08f1ac32-f49e-4aa2-9ab8-8a1aac7bef36-agent-certs" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.524154 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d4aae53-afc9-4373-953d-b9b7c323e3ee" volumeName="kubernetes.io/projected/1d4aae53-afc9-4373-953d-b9b7c323e3ee-kube-api-access-tvgz2" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.524172 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c877970-90d9-43ba-9c9b-4a22fbda1dc0" volumeName="kubernetes.io/configmap/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-cni-binary-copy" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.524191 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3341bf47-8c8f-4073-8758-122e04e0b138" volumeName="kubernetes.io/secret/3341bf47-8c8f-4073-8758-122e04e0b138-original-pull-secret" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.524204 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70c3c6bc-c455-4fe6-9099-1ceb92abeaf0" volumeName="kubernetes.io/projected/70c3c6bc-c455-4fe6-9099-1ceb92abeaf0-kube-api-access-gfs8x" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.524270 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f285a424-0cc2-41b8-99b9-92db326ea01d" volumeName="kubernetes.io/configmap/f285a424-0cc2-41b8-99b9-92db326ea01d-trusted-ca-bundle" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.524293 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f285a424-0cc2-41b8-99b9-92db326ea01d" volumeName="kubernetes.io/empty-dir/f285a424-0cc2-41b8-99b9-92db326ea01d-archives-path" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.524305 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05d96464-f946-4a83-88e5-ececfee30b4c" volumeName="kubernetes.io/empty-dir/05d96464-f946-4a83-88e5-ececfee30b4c-tmp-dir" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.524415 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17696d35-744c-434c-94de-33ec719feac9" volumeName="kubernetes.io/empty-dir/17696d35-744c-434c-94de-33ec719feac9-data-volume" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.524442 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17696d35-744c-434c-94de-33ec719feac9" volumeName="kubernetes.io/projected/17696d35-744c-434c-94de-33ec719feac9-kube-api-access-5ghv7" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.524455 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18d4542b-53c1-473f-9605-c82fd5a642f4" volumeName="kubernetes.io/projected/18d4542b-53c1-473f-9605-c82fd5a642f4-kube-api-access-k52xf" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.524467 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18d4542b-53c1-473f-9605-c82fd5a642f4" volumeName="kubernetes.io/secret/18d4542b-53c1-473f-9605-c82fd5a642f4-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.524489 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c877970-90d9-43ba-9c9b-4a22fbda1dc0" volumeName="kubernetes.io/projected/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-kube-api-access-k8snd" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.524504 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18d4542b-53c1-473f-9605-c82fd5a642f4" volumeName="kubernetes.io/configmap/18d4542b-53c1-473f-9605-c82fd5a642f4-env-overrides" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.524522 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19538cda-3ba8-4bad-bb47-23a2511736e7" volumeName="kubernetes.io/projected/19538cda-3ba8-4bad-bb47-23a2511736e7-kube-api-access-bsrnm" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.524599 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d4aae53-afc9-4373-953d-b9b7c323e3ee" volumeName="kubernetes.io/configmap/1d4aae53-afc9-4373-953d-b9b7c323e3ee-iptables-alerter-script" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.524611 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25ca3b5e-7eba-4411-be09-5da0fe41e473" volumeName="kubernetes.io/configmap/25ca3b5e-7eba-4411-be09-5da0fe41e473-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.524630 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4e3a1e7-fed8-4969-b9ae-baec7c2dade8" volumeName="kubernetes.io/empty-dir/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-tmp" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.524734 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4e3a1e7-fed8-4969-b9ae-baec7c2dade8" volumeName="kubernetes.io/empty-dir/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-tuned" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.524777 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4e3a1e7-fed8-4969-b9ae-baec7c2dade8" volumeName="kubernetes.io/projected/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-kube-api-access-hl9th" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.524793 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08f1ac32-f49e-4aa2-9ab8-8a1aac7bef36" volumeName="kubernetes.io/configmap/08f1ac32-f49e-4aa2-9ab8-8a1aac7bef36-konnectivity-ca" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.524810 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17696d35-744c-434c-94de-33ec719feac9" volumeName="kubernetes.io/secret/17696d35-744c-434c-94de-33ec719feac9-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.524828 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f285a424-0cc2-41b8-99b9-92db326ea01d" volumeName="kubernetes.io/projected/f285a424-0cc2-41b8-99b9-92db326ea01d-kube-api-access-gl5cb" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.524841 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18d4542b-53c1-473f-9605-c82fd5a642f4" volumeName="kubernetes.io/configmap/18d4542b-53c1-473f-9605-c82fd5a642f4-ovnkube-config" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.524863 4560 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d528ee4-1b51-4fdb-9166-56f14060d6af" volumeName="kubernetes.io/projected/3d528ee4-1b51-4fdb-9166-56f14060d6af-kube-api-access-kbrm8" seLinuxMountContext="" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.524874 4560 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:13:44.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.524883 4560 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:13:44.566768 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.566731 4560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:13:44.568250 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.568224 4560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:13:44.568250 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.568249 4560 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:13:44.568442 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.568271 4560 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:13:44.568442 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:13:44.568311 4560 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 21:13:44.597455 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.597432 4560 manager.go:324] Recovery completed Oct 30 21:13:44.603173 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.603149 4560 generic.go:356] "Generic (PLEG): container finished" podID="25ca3b5e-7eba-4411-be09-5da0fe41e473" containerID="0d5063d0b3ed59c57bcba7cc2703a10e8b1c74c180897339c5b7a92896f5fff3" exitCode=0 Oct 30 21:13:44.603173 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.603170 4560 generic.go:356] "Generic (PLEG): container finished" podID="25ca3b5e-7eba-4411-be09-5da0fe41e473" containerID="9a7acce348aa6e445a497c0abb97cbfa12fd210b0f1492b8a29eabc8869c4a5a" exitCode=0 Oct 30 21:13:44.603173 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.603176 4560 generic.go:356] "Generic (PLEG): container finished" podID="25ca3b5e-7eba-4411-be09-5da0fe41e473" containerID="d5ec71db208910cf6d3cd5435aa31d642c799a5bf099d6db2e55e4cca269192c" exitCode=0 Oct 30 21:13:44.603390 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.603182 4560 generic.go:356] "Generic (PLEG): container finished" podID="25ca3b5e-7eba-4411-be09-5da0fe41e473" containerID="ca0c856cf5a71b0d0dffe7d1a48d5b3f4015b2350ae5c82fe0f7e172d80c1084" exitCode=0 Oct 30 21:13:44.603390 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.603187 4560 generic.go:356] "Generic (PLEG): container finished" podID="25ca3b5e-7eba-4411-be09-5da0fe41e473" containerID="db12be8606824c372929e38943cb47e7ba707b5451d7a7e763d9c780fbc675a1" exitCode=0 Oct 30 21:13:44.603390 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.603193 4560 generic.go:356] "Generic (PLEG): container finished" podID="25ca3b5e-7eba-4411-be09-5da0fe41e473" containerID="78406c1ea5663373189fa90efef3f9fe30b11084885bc25ee2e459f7aebb2718" exitCode=0 Oct 30 21:13:44.606137 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.606109 4560 generic.go:356] "Generic (PLEG): container finished" podID="2e151e7c539d69d81e686cdbfb721e69" containerID="20aed17bbb7fa67e394737034e85203c006f8e6d156a208c62d502d17a1bbe4b" exitCode=0 Oct 30 21:13:44.614282 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.614264 4560 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:13:44.615228 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.615214 4560 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:13:44.615298 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.615242 4560 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:13:44.615298 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.615255 4560 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:13:44.617130 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.617113 4560 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:13:44.617130 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.617125 4560 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:13:44.617130 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.617129 4560 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:13:44.617305 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.617149 4560 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:13:44.617371 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.617304 4560 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 21:13:44.617371 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.617316 4560 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 21:13:44.617465 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.617402 4560 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 21:13:44.617465 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.617410 4560 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 21:13:44.617465 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.617418 4560 policy_none.go:50] "Start" Oct 30 21:13:44.617465 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.617429 4560 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:13:44.617465 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.617443 4560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:13:44.617678 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.617618 4560 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 21:13:44.617678 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.617628 4560 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 21:13:44.617678 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.617637 4560 policy_none.go:44] "Start" Oct 30 21:13:44.617979 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.617959 4560 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:13:44.618085 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.617987 4560 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:13:44.618085 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.618001 4560 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:13:44.621761 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:13:44.621743 4560 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-83.us-west-2.compute.internal\" not found" Oct 30 21:13:44.621839 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.621823 4560 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:13:44.625214 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.625201 4560 manager.go:341] "Starting Device Plugin manager" Oct 30 21:13:44.625291 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:13:44.625223 4560 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:13:44.625291 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.625231 4560 server.go:87] "Starting device plugin registration server" Oct 30 21:13:44.625532 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.625520 4560 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:13:44.625586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.625536 4560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:13:44.625658 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.625642 4560 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:13:44.625766 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.625734 4560 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:13:44.625823 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.625771 4560 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:13:44.625823 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.625777 4560 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:13:44.625912 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.625827 4560 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:13:44.625761919Z","UUID":"0d76092a-fb70-4a28-8164-2957498303a6","Handler":null,"Name":"","Endpoint":""} Oct 30 21:13:44.627258 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:13:44.627207 4560 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:13:44.627258 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:13:44.627244 4560 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-83.us-west-2.compute.internal\" not found" Oct 30 21:13:44.627390 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.627377 4560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:13:44.627422 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.627402 4560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:13:44.654866 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.654846 4560 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:13:44.669317 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.669287 4560 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-83.us-west-2.compute.internal"] Oct 30 21:13:44.669421 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.669409 4560 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:13:44.670205 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.670182 4560 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:13:44.670275 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.670212 4560 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:13:44.670275 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.670224 4560 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:13:44.670496 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.670485 4560 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:13:44.670620 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.670605 4560 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:13:44.671213 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.671200 4560 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:13:44.671213 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.671205 4560 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:13:44.671328 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.671228 4560 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:13:44.671328 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.671229 4560 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:13:44.671328 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.671255 4560 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:13:44.671328 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.671240 4560 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:13:44.671530 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:13:44.671467 4560 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-83.us-west-2.compute.internal\" not found" node="ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:44.671589 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.671536 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-83.us-west-2.compute.internal" event={"ID":"8b9addd42169977c49b17223d2454db0","Type":"ContainerStarted","Data":"3d1dd588a97dc824f48fd65376ffd16339e7276d35ce8687df74047bd3fc2a91"} Oct 30 21:13:44.671589 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.671581 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-83.us-west-2.compute.internal" event={"ID":"8b9addd42169977c49b17223d2454db0","Type":"ContainerStarted","Data":"81378ec2ad2c76d0f97979360af798bf2d824bde7db7842663a8373b224f5bb0"} Oct 30 21:13:44.671679 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.671634 4560 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:13:44.671726 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.671685 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal" event={"ID":"2e151e7c539d69d81e686cdbfb721e69","Type":"ContainerStarted","Data":"6bb4891ae70f10c63db27dd956f9377bebab1778b0eef09d1d00221b23763a08"} Oct 30 21:13:44.671726 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.671706 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal" event={"ID":"2e151e7c539d69d81e686cdbfb721e69","Type":"ContainerDied","Data":"20aed17bbb7fa67e394737034e85203c006f8e6d156a208c62d502d17a1bbe4b"} Oct 30 21:13:44.671809 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.671724 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal" event={"ID":"2e151e7c539d69d81e686cdbfb721e69","Type":"ContainerStarted","Data":"0bc90de4c74ac62133ee2f84e8d2364207b45a0853fc7b80529262f8213ce9c2"} Oct 30 21:13:44.673956 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.672613 4560 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:13:44.673956 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.672645 4560 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:13:44.673956 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.672661 4560 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:13:44.673956 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:13:44.672820 4560 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-83.us-west-2.compute.internal\" not found" node="ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:44.701427 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.701398 4560 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 21:13:44.701538 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.701482 4560 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:13:44.726289 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.726272 4560 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:13:44.727042 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.727026 4560 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:13:44.727106 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.727054 4560 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:13:44.727106 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.727065 4560 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:13:44.727106 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.727092 4560 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:44.748627 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.748609 4560 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:13:44.789199 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.789174 4560 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:13:44.802373 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.802331 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e151e7c539d69d81e686cdbfb721e69-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal\" (UID: \"2e151e7c539d69d81e686cdbfb721e69\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:44.802463 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.802427 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e151e7c539d69d81e686cdbfb721e69-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal\" (UID: \"2e151e7c539d69d81e686cdbfb721e69\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:44.802463 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.802452 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b9addd42169977c49b17223d2454db0-config\") pod \"kube-apiserver-proxy-ip-10-0-129-83.us-west-2.compute.internal\" (UID: \"8b9addd42169977c49b17223d2454db0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:44.821461 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.821435 4560 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:44.838469 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.838448 4560 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:44.838548 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.838501 4560 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:44.838999 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.838982 4560 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeNotReady" Oct 30 21:13:44.839060 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.839011 4560 setters.go:546] "Node became not ready" node="ip-10-0-129-83.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-30T21:13:44Z","lastTransitionTime":"2026-10-30T21:13:44Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Oct 30 21:13:44.857253 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.857228 4560 nodeinfomanager.go:471] existing CSINode "ip-10-0-129-83.us-west-2.compute.internal" is owned by different node (oldNodeID="4f9f1775-2fec-433f-93e9-3c8f5c7dc48c", newNodeID=""), cleaning up... Oct 30 21:13:44.883357 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.883282 4560 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:13:44.883357 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:13:44.883326 4560 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:44.883357 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.883349 4560 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:44.902839 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.902812 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e151e7c539d69d81e686cdbfb721e69-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal\" (UID: \"2e151e7c539d69d81e686cdbfb721e69\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:44.902951 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.902869 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b9addd42169977c49b17223d2454db0-config\") pod \"kube-apiserver-proxy-ip-10-0-129-83.us-west-2.compute.internal\" (UID: \"8b9addd42169977c49b17223d2454db0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:44.902951 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.902886 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e151e7c539d69d81e686cdbfb721e69-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal\" (UID: \"2e151e7c539d69d81e686cdbfb721e69\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:44.902951 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.902907 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e151e7c539d69d81e686cdbfb721e69-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal\" (UID: \"2e151e7c539d69d81e686cdbfb721e69\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:44.902951 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.902886 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b9addd42169977c49b17223d2454db0-config\") pod \"kube-apiserver-proxy-ip-10-0-129-83.us-west-2.compute.internal\" (UID: \"8b9addd42169977c49b17223d2454db0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:44.902951 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.902934 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e151e7c539d69d81e686cdbfb721e69-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal\" (UID: \"2e151e7c539d69d81e686cdbfb721e69\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:44.938642 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:44.938623 4560 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:13:44.938730 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:13:44.938658 4560 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-83.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:45.502678 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.502652 4560 apiserver.go:52] "Watching apiserver" Oct 30 21:13:45.601568 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.601542 4560 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:13:45.602147 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.602118 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-jbjqt","openshift-multus/multus-additional-cni-plugins-brt6h","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2","openshift-dns/node-resolver-p4c9m","openshift-ingress-canary/ingress-canary-2kqn4","kube-system/global-pull-secret-syncer-jztdg","openshift-cluster-node-tuning-operator/tuned-pgnjv","openshift-insights/insights-runtime-extractor-hq4sb","openshift-multus/multus-zn2wp","kube-system/kube-apiserver-proxy-ip-10-0-129-83.us-west-2.compute.internal","openshift-image-registry/node-ca-q8g7b","openshift-network-operator/iptables-alerter-jfprx","openshift-dns/dns-default-spbvd","openshift-network-diagnostics/network-check-target-lw64d","openshift-service-ca/service-ca-5fffc97bdf-zxlqv","openshift-multus/network-metrics-daemon-n4g68","kube-system/konnectivity-agent-vpxpj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal","openshift-insights/periodic-gathering-89mnx-pncws","openshift-kube-storage-version-migrator/migrator-66df5d496b-hz5dc"] Oct 30 21:13:45.607031 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.606478 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4g68" Oct 30 21:13:45.610049 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.610020 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-spbvd" Oct 30 21:13:45.612574 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.612556 4560 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:45.612725 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.612709 4560 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:45.622716 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.622700 4560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:13:45.653209 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.653184 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:13:45.653209 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.653196 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-x6zwr\"" Oct 30 21:13:45.653308 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.653236 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qzjzq\"" Oct 30 21:13:45.653308 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.653271 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4v9xp\"" Oct 30 21:13:45.654693 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.654677 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:13:45.654693 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.654690 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:13:45.654847 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.654681 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:13:45.655503 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.655467 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:13:45.655503 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.655473 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ph6zc\"" Oct 30 21:13:45.655503 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.655484 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:13:45.655662 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.655508 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-h52wm\"" Oct 30 21:13:45.655662 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.655524 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:13:45.655662 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.655545 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:13:45.655662 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.655603 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:13:45.655662 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.655638 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:13:45.655797 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.655666 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:13:45.657540 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.657527 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:13:45.657799 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.657785 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:13:45.657829 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.657798 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:13:45.657829 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.657787 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:13:45.658047 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.658036 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:13:45.658076 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.658057 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:13:45.658847 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.658833 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:13:45.659088 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.659076 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:13:45.659140 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.659128 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:13:45.659418 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.659407 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:13:45.659463 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.659454 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-dpfbq\"" Oct 30 21:13:45.659886 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.659875 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-79sqh\"" Oct 30 21:13:45.659934 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.659924 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:13:45.660110 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.660094 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:13:45.660147 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.660131 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w5d85\"" Oct 30 21:13:45.660147 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.660138 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:13:45.660147 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.660143 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-q4s27\"" Oct 30 21:13:45.660223 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.660159 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sqpcg\"" Oct 30 21:13:45.660695 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.660683 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:13:45.699737 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.699718 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:13:45.700033 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.700016 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kvf2h\"" Oct 30 21:13:45.700372 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.700358 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:13:45.700485 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.700468 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:13:45.700559 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.700495 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-5fn5j\"" Oct 30 21:13:45.700559 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.700503 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:13:45.700559 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.700531 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:13:45.700651 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.700585 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:13:45.700769 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.700754 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 21:13:45.700769 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.700758 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:13:45.700839 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.700773 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:13:45.701000 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.700988 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:13:45.701044 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.701012 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:13:45.701044 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.701024 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:13:45.701044 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.701029 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 21:13:45.701130 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.701035 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 21:13:45.701372 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.701334 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-nvvjf\"" Oct 30 21:13:45.701372 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.701357 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:13:45.701538 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.701380 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:13:45.701538 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.701404 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:13:45.701538 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.701409 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-lm9pr\"" Oct 30 21:13:45.701538 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.701428 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b9xqh\"" Oct 30 21:13:45.701538 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.701469 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-2smgj\"" Oct 30 21:13:45.701736 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.701552 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-9zv66\"" Oct 30 21:13:45.701736 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.701555 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 21:13:45.701736 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.701566 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hwnd5\"" Oct 30 21:13:45.701736 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.701585 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:13:45.702552 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.702532 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 21:13:45.702552 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.702542 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 21:13:45.704416 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.704404 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 21:13:45.706438 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.706422 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/25ca3b5e-7eba-4411-be09-5da0fe41e473-tuning-conf-dir\") pod \"multus-additional-cni-plugins-brt6h\" (UID: \"25ca3b5e-7eba-4411-be09-5da0fe41e473\") " pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:45.706489 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.706457 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05d96464-f946-4a83-88e5-ececfee30b4c-tmp-dir\") pod \"node-resolver-p4c9m\" (UID: \"05d96464-f946-4a83-88e5-ececfee30b4c\") " pod="openshift-dns/node-resolver-p4c9m" Oct 30 21:13:45.706489 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.706480 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3341bf47-8c8f-4073-8758-122e04e0b138-original-pull-secret\") pod \"global-pull-secret-syncer-jztdg\" (UID: \"3341bf47-8c8f-4073-8758-122e04e0b138\") " pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:45.706564 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.706539 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-sysctl-conf\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:45.706601 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.706575 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-host-run-k8s-cni-cncf-io\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.706647 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.706604 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-etc-kubernetes\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.706647 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.706632 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54b8c0a5-e1a7-4d02-a402-b3cadc139e28-host\") pod \"node-ca-q8g7b\" (UID: \"54b8c0a5-e1a7-4d02-a402-b3cadc139e28\") " pod="openshift-image-registry/node-ca-q8g7b" Oct 30 21:13:45.706792 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.706652 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 21:13:45.706792 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.706661 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvbqm\" (UniqueName: \"kubernetes.io/projected/25ca3b5e-7eba-4411-be09-5da0fe41e473-kube-api-access-mvbqm\") pod \"multus-additional-cni-plugins-brt6h\" (UID: \"25ca3b5e-7eba-4411-be09-5da0fe41e473\") " pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:45.706792 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.706684 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9597db44-1ca0-4fa8-ade7-cb55eeca9593-etc-selinux\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:45.706792 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.706694 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05d96464-f946-4a83-88e5-ececfee30b4c-tmp-dir\") pod \"node-resolver-p4c9m\" (UID: \"05d96464-f946-4a83-88e5-ececfee30b4c\") " pod="openshift-dns/node-resolver-p4c9m" Oct 30 21:13:45.706792 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.706719 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsrnm\" (UniqueName: \"kubernetes.io/projected/19538cda-3ba8-4bad-bb47-23a2511736e7-kube-api-access-bsrnm\") pod \"network-check-target-lw64d\" (UID: \"19538cda-3ba8-4bad-bb47-23a2511736e7\") " pod="openshift-network-diagnostics/network-check-target-lw64d" Oct 30 21:13:45.706792 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.706745 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f96e4d00-99fb-4196-a1a9-83599bfd875f-signing-cabundle\") pod \"service-ca-5fffc97bdf-zxlqv\" (UID: \"f96e4d00-99fb-4196-a1a9-83599bfd875f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zxlqv" Oct 30 21:13:45.707002 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.706828 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3341bf47-8c8f-4073-8758-122e04e0b138-original-pull-secret\") pod \"global-pull-secret-syncer-jztdg\" (UID: \"3341bf47-8c8f-4073-8758-122e04e0b138\") " pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:45.707002 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.706848 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-host-run-netns\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.707002 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.706929 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-cni-netd\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.707002 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.706972 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/18d4542b-53c1-473f-9605-c82fd5a642f4-env-overrides\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.707135 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.707010 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/25ca3b5e-7eba-4411-be09-5da0fe41e473-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-brt6h\" (UID: \"25ca3b5e-7eba-4411-be09-5da0fe41e473\") " pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:45.707135 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.707037 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f96e4d00-99fb-4196-a1a9-83599bfd875f-signing-cabundle\") pod \"service-ca-5fffc97bdf-zxlqv\" (UID: \"f96e4d00-99fb-4196-a1a9-83599bfd875f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zxlqv" Oct 30 21:13:45.707135 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.707108 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/08f1ac32-f49e-4aa2-9ab8-8a1aac7bef36-konnectivity-ca\") pod \"konnectivity-agent-vpxpj\" (UID: \"08f1ac32-f49e-4aa2-9ab8-8a1aac7bef36\") " pod="kube-system/konnectivity-agent-vpxpj" Oct 30 21:13:45.707225 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.707172 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/17696d35-744c-434c-94de-33ec719feac9-crio-socket\") pod \"insights-runtime-extractor-hq4sb\" (UID: \"17696d35-744c-434c-94de-33ec719feac9\") " pod="openshift-insights/insights-runtime-extractor-hq4sb" Oct 30 21:13:45.707225 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.707196 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/18d4542b-53c1-473f-9605-c82fd5a642f4-env-overrides\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.707225 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.707207 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/17696d35-744c-434c-94de-33ec719feac9-data-volume\") pod \"insights-runtime-extractor-hq4sb\" (UID: \"17696d35-744c-434c-94de-33ec719feac9\") " pod="openshift-insights/insights-runtime-extractor-hq4sb" Oct 30 21:13:45.707315 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.707264 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/17696d35-744c-434c-94de-33ec719feac9-data-volume\") pod \"insights-runtime-extractor-hq4sb\" (UID: \"17696d35-744c-434c-94de-33ec719feac9\") " pod="openshift-insights/insights-runtime-extractor-hq4sb" Oct 30 21:13:45.707315 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.707264 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9597db44-1ca0-4fa8-ade7-cb55eeca9593-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:45.707315 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.707291 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/08f1ac32-f49e-4aa2-9ab8-8a1aac7bef36-konnectivity-ca\") pod \"konnectivity-agent-vpxpj\" (UID: \"08f1ac32-f49e-4aa2-9ab8-8a1aac7bef36\") " pod="kube-system/konnectivity-agent-vpxpj" Oct 30 21:13:45.707446 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.707299 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4eed118d-1567-4be4-8c90-5dc0e7478827-config-volume\") pod \"dns-default-spbvd\" (UID: \"4eed118d-1567-4be4-8c90-5dc0e7478827\") " pod="openshift-dns/dns-default-spbvd" Oct 30 21:13:45.707446 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.707362 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdp4m\" (UniqueName: \"kubernetes.io/projected/f96e4d00-99fb-4196-a1a9-83599bfd875f-kube-api-access-kdp4m\") pod \"service-ca-5fffc97bdf-zxlqv\" (UID: \"f96e4d00-99fb-4196-a1a9-83599bfd875f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zxlqv" Oct 30 21:13:45.707446 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.707400 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/25ca3b5e-7eba-4411-be09-5da0fe41e473-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-brt6h\" (UID: \"25ca3b5e-7eba-4411-be09-5da0fe41e473\") " pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:45.707558 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.707440 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-log-socket\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.707558 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.707479 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg66z\" (UniqueName: \"kubernetes.io/projected/54b8c0a5-e1a7-4d02-a402-b3cadc139e28-kube-api-access-jg66z\") pod \"node-ca-q8g7b\" (UID: \"54b8c0a5-e1a7-4d02-a402-b3cadc139e28\") " pod="openshift-image-registry/node-ca-q8g7b" Oct 30 21:13:45.707558 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.707539 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4eed118d-1567-4be4-8c90-5dc0e7478827-config-volume\") pod \"dns-default-spbvd\" (UID: \"4eed118d-1567-4be4-8c90-5dc0e7478827\") " pod="openshift-dns/dns-default-spbvd" Oct 30 21:13:45.707719 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.707597 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9597db44-1ca0-4fa8-ade7-cb55eeca9593-sys-fs\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:45.707719 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.707629 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3341bf47-8c8f-4073-8758-122e04e0b138-kubelet-config\") pod \"global-pull-secret-syncer-jztdg\" (UID: \"3341bf47-8c8f-4073-8758-122e04e0b138\") " pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:45.707719 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.707657 4560 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:13:45.707719 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.707665 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-tuned\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:45.707719 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.707688 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30bde564-b785-4359-9377-ffec880fb18c-metrics-certs\") pod \"network-metrics-daemon-n4g68\" (UID: \"30bde564-b785-4359-9377-ffec880fb18c\") " pod="openshift-multus/network-metrics-daemon-n4g68" Oct 30 21:13:45.707719 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:13:45.707698 4560 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-83.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:45.707719 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.707703 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-run-systemd\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.707966 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.707722 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-modprobe-d\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:45.707966 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.707733 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-tuned\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:45.707966 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.707770 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-sysconfig\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:45.707966 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.707841 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-lib-modules\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:45.707966 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.707935 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-cnibin\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.708116 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.708019 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-host-var-lib-cni-bin\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.708116 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.708050 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-host-run-multus-certs\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.708232 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.708107 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/18d4542b-53c1-473f-9605-c82fd5a642f4-ovnkube-script-lib\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.708232 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.708139 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k52xf\" (UniqueName: \"kubernetes.io/projected/18d4542b-53c1-473f-9605-c82fd5a642f4-kube-api-access-k52xf\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.708232 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.708159 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbrm8\" (UniqueName: \"kubernetes.io/projected/3d528ee4-1b51-4fdb-9166-56f14060d6af-kube-api-access-kbrm8\") pod \"ingress-canary-2kqn4\" (UID: \"3d528ee4-1b51-4fdb-9166-56f14060d6af\") " pod="openshift-ingress-canary/ingress-canary-2kqn4" Oct 30 21:13:45.708232 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.708180 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl5cb\" (UniqueName: \"kubernetes.io/projected/f285a424-0cc2-41b8-99b9-92db326ea01d-kube-api-access-gl5cb\") pod \"periodic-gathering-89mnx-pncws\" (UID: \"f285a424-0cc2-41b8-99b9-92db326ea01d\") " pod="openshift-insights/periodic-gathering-89mnx-pncws" Oct 30 21:13:45.708232 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.708205 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-system-cni-dir\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.708232 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.708230 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-multus-cni-dir\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.708572 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.708253 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-systemd-units\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.708572 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.708278 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25ca3b5e-7eba-4411-be09-5da0fe41e473-system-cni-dir\") pod \"multus-additional-cni-plugins-brt6h\" (UID: \"25ca3b5e-7eba-4411-be09-5da0fe41e473\") " pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:45.708572 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.708304 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1d4aae53-afc9-4373-953d-b9b7c323e3ee-iptables-alerter-script\") pod \"iptables-alerter-jfprx\" (UID: \"1d4aae53-afc9-4373-953d-b9b7c323e3ee\") " pod="openshift-network-operator/iptables-alerter-jfprx" Oct 30 21:13:45.708572 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.708367 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/18d4542b-53c1-473f-9605-c82fd5a642f4-ovnkube-script-lib\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.708572 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.708373 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d528ee4-1b51-4fdb-9166-56f14060d6af-cert\") pod \"ingress-canary-2kqn4\" (UID: \"3d528ee4-1b51-4fdb-9166-56f14060d6af\") " pod="openshift-ingress-canary/ingress-canary-2kqn4" Oct 30 21:13:45.708572 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.708488 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f285a424-0cc2-41b8-99b9-92db326ea01d-trusted-ca-bundle\") pod \"periodic-gathering-89mnx-pncws\" (UID: \"f285a424-0cc2-41b8-99b9-92db326ea01d\") " pod="openshift-insights/periodic-gathering-89mnx-pncws" Oct 30 21:13:45.708572 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.708535 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f96e4d00-99fb-4196-a1a9-83599bfd875f-signing-key\") pod \"service-ca-5fffc97bdf-zxlqv\" (UID: \"f96e4d00-99fb-4196-a1a9-83599bfd875f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zxlqv" Oct 30 21:13:45.708799 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.708602 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1d4aae53-afc9-4373-953d-b9b7c323e3ee-iptables-alerter-script\") pod \"iptables-alerter-jfprx\" (UID: \"1d4aae53-afc9-4373-953d-b9b7c323e3ee\") " pod="openshift-network-operator/iptables-alerter-jfprx" Oct 30 21:13:45.708941 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.708910 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-multus-daemon-config\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.708993 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.708949 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d528ee4-1b51-4fdb-9166-56f14060d6af-cert\") pod \"ingress-canary-2kqn4\" (UID: \"3d528ee4-1b51-4fdb-9166-56f14060d6af\") " pod="openshift-ingress-canary/ingress-canary-2kqn4" Oct 30 21:13:45.709031 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.709009 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-node-log\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.709031 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.709019 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f96e4d00-99fb-4196-a1a9-83599bfd875f-signing-key\") pod \"service-ca-5fffc97bdf-zxlqv\" (UID: \"f96e4d00-99fb-4196-a1a9-83599bfd875f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zxlqv" Oct 30 21:13:45.709126 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.709037 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-run-ovn-kubernetes\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.709126 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.709082 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfs8x\" (UniqueName: \"kubernetes.io/projected/70c3c6bc-c455-4fe6-9099-1ceb92abeaf0-kube-api-access-gfs8x\") pod \"migrator-66df5d496b-hz5dc\" (UID: \"70c3c6bc-c455-4fe6-9099-1ceb92abeaf0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hz5dc" Oct 30 21:13:45.710586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.709251 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9597db44-1ca0-4fa8-ade7-cb55eeca9593-socket-dir\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:45.710586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.709251 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f285a424-0cc2-41b8-99b9-92db326ea01d-trusted-ca-bundle\") pod \"periodic-gathering-89mnx-pncws\" (UID: \"f285a424-0cc2-41b8-99b9-92db326ea01d\") " pod="openshift-insights/periodic-gathering-89mnx-pncws" Oct 30 21:13:45.710586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.709302 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-multus-daemon-config\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.710586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.709317 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f285a424-0cc2-41b8-99b9-92db326ea01d-archives-path\") pod \"periodic-gathering-89mnx-pncws\" (UID: \"f285a424-0cc2-41b8-99b9-92db326ea01d\") " pod="openshift-insights/periodic-gathering-89mnx-pncws" Oct 30 21:13:45.710586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.709370 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eed118d-1567-4be4-8c90-5dc0e7478827-tmp-dir\") pod \"dns-default-spbvd\" (UID: \"4eed118d-1567-4be4-8c90-5dc0e7478827\") " pod="openshift-dns/dns-default-spbvd" Oct 30 21:13:45.710586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.709401 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f285a424-0cc2-41b8-99b9-92db326ea01d-archives-path\") pod \"periodic-gathering-89mnx-pncws\" (UID: \"f285a424-0cc2-41b8-99b9-92db326ea01d\") " pod="openshift-insights/periodic-gathering-89mnx-pncws" Oct 30 21:13:45.710586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.709405 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk4wl\" (UniqueName: \"kubernetes.io/projected/4eed118d-1567-4be4-8c90-5dc0e7478827-kube-api-access-tk4wl\") pod \"dns-default-spbvd\" (UID: \"4eed118d-1567-4be4-8c90-5dc0e7478827\") " pod="openshift-dns/dns-default-spbvd" Oct 30 21:13:45.710586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.709438 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-etc-openvswitch\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.710586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.709460 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eed118d-1567-4be4-8c90-5dc0e7478827-tmp-dir\") pod \"dns-default-spbvd\" (UID: \"4eed118d-1567-4be4-8c90-5dc0e7478827\") " pod="openshift-dns/dns-default-spbvd" Oct 30 21:13:45.710586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.709471 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.710586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.709539 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/18d4542b-53c1-473f-9605-c82fd5a642f4-ovn-node-metrics-cert\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.710586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.709582 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ghv7\" (UniqueName: \"kubernetes.io/projected/17696d35-744c-434c-94de-33ec719feac9-kube-api-access-5ghv7\") pod \"insights-runtime-extractor-hq4sb\" (UID: \"17696d35-744c-434c-94de-33ec719feac9\") " pod="openshift-insights/insights-runtime-extractor-hq4sb" Oct 30 21:13:45.710586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.709615 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f285a424-0cc2-41b8-99b9-92db326ea01d-service-ca-bundle\") pod \"periodic-gathering-89mnx-pncws\" (UID: \"f285a424-0cc2-41b8-99b9-92db326ea01d\") " pod="openshift-insights/periodic-gathering-89mnx-pncws" Oct 30 21:13:45.710586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.709692 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-kubernetes\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:45.710586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.709726 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-sys\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:45.710586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.709763 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/08f1ac32-f49e-4aa2-9ab8-8a1aac7bef36-agent-certs\") pod \"konnectivity-agent-vpxpj\" (UID: \"08f1ac32-f49e-4aa2-9ab8-8a1aac7bef36\") " pod="kube-system/konnectivity-agent-vpxpj" Oct 30 21:13:45.710586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.709768 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f285a424-0cc2-41b8-99b9-92db326ea01d-service-ca-bundle\") pod \"periodic-gathering-89mnx-pncws\" (UID: \"f285a424-0cc2-41b8-99b9-92db326ea01d\") " pod="openshift-insights/periodic-gathering-89mnx-pncws" Oct 30 21:13:45.710586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.709791 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvgz2\" (UniqueName: \"kubernetes.io/projected/1d4aae53-afc9-4373-953d-b9b7c323e3ee-kube-api-access-tvgz2\") pod \"iptables-alerter-jfprx\" (UID: \"1d4aae53-afc9-4373-953d-b9b7c323e3ee\") " pod="openshift-network-operator/iptables-alerter-jfprx" Oct 30 21:13:45.710586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.709816 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/18d4542b-53c1-473f-9605-c82fd5a642f4-ovn-node-metrics-cert\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.710586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.709829 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17696d35-744c-434c-94de-33ec719feac9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hq4sb\" (UID: \"17696d35-744c-434c-94de-33ec719feac9\") " pod="openshift-insights/insights-runtime-extractor-hq4sb" Oct 30 21:13:45.710586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.709953 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cctxz\" (UniqueName: \"kubernetes.io/projected/05d96464-f946-4a83-88e5-ececfee30b4c-kube-api-access-cctxz\") pod \"node-resolver-p4c9m\" (UID: \"05d96464-f946-4a83-88e5-ececfee30b4c\") " pod="openshift-dns/node-resolver-p4c9m" Oct 30 21:13:45.710586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.710006 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-multus-socket-dir-parent\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.710586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.710082 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/08f1ac32-f49e-4aa2-9ab8-8a1aac7bef36-agent-certs\") pod \"konnectivity-agent-vpxpj\" (UID: \"08f1ac32-f49e-4aa2-9ab8-8a1aac7bef36\") " pod="kube-system/konnectivity-agent-vpxpj" Oct 30 21:13:45.710586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.710102 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8snd\" (UniqueName: \"kubernetes.io/projected/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-kube-api-access-k8snd\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.710586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.710133 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/25ca3b5e-7eba-4411-be09-5da0fe41e473-cni-binary-copy\") pod \"multus-additional-cni-plugins-brt6h\" (UID: \"25ca3b5e-7eba-4411-be09-5da0fe41e473\") " pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:45.710586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.710180 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-tmp\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:45.710586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.710213 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-cni-binary-copy\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.710586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.710245 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8884\" (UniqueName: \"kubernetes.io/projected/30bde564-b785-4359-9377-ffec880fb18c-kube-api-access-l8884\") pod \"network-metrics-daemon-n4g68\" (UID: \"30bde564-b785-4359-9377-ffec880fb18c\") " pod="openshift-multus/network-metrics-daemon-n4g68" Oct 30 21:13:45.710586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.710279 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-run-openvswitch\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.710586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.710304 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-run-ovn\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.710586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.710333 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-tmp\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:45.710586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.710332 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-cni-bin\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.710586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.710358 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17696d35-744c-434c-94de-33ec719feac9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hq4sb\" (UID: \"17696d35-744c-434c-94de-33ec719feac9\") " pod="openshift-insights/insights-runtime-extractor-hq4sb" Oct 30 21:13:45.710586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.710362 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/25ca3b5e-7eba-4411-be09-5da0fe41e473-cni-binary-copy\") pod \"multus-additional-cni-plugins-brt6h\" (UID: \"25ca3b5e-7eba-4411-be09-5da0fe41e473\") " pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:45.710586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.710393 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17696d35-744c-434c-94de-33ec719feac9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hq4sb\" (UID: \"17696d35-744c-434c-94de-33ec719feac9\") " pod="openshift-insights/insights-runtime-extractor-hq4sb" Oct 30 21:13:45.710586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.710511 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-run\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:45.710586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.710606 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-cni-binary-copy\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.712179 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.710670 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-run-netns\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.712179 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.710691 4560 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:13:45.712179 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.710703 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/25ca3b5e-7eba-4411-be09-5da0fe41e473-cnibin\") pod \"multus-additional-cni-plugins-brt6h\" (UID: \"25ca3b5e-7eba-4411-be09-5da0fe41e473\") " pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:45.712179 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.710734 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/25ca3b5e-7eba-4411-be09-5da0fe41e473-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-brt6h\" (UID: \"25ca3b5e-7eba-4411-be09-5da0fe41e473\") " pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:45.712179 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:13:45.710754 4560 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-83.us-west-2.compute.internal" Oct 30 21:13:45.712179 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.710765 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9597db44-1ca0-4fa8-ade7-cb55eeca9593-device-dir\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:45.712179 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.710796 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl9th\" (UniqueName: \"kubernetes.io/projected/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-kube-api-access-hl9th\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:45.712179 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.710898 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-host-var-lib-kubelet\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.712179 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.710937 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-slash\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.712179 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.710986 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-var-lib-openvswitch\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.712179 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.711009 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/25ca3b5e-7eba-4411-be09-5da0fe41e473-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-brt6h\" (UID: \"25ca3b5e-7eba-4411-be09-5da0fe41e473\") " pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:45.712179 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.711015 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-host\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:45.712179 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.711278 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-kubelet\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.712179 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.711321 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3341bf47-8c8f-4073-8758-122e04e0b138-dbus\") pod \"global-pull-secret-syncer-jztdg\" (UID: \"3341bf47-8c8f-4073-8758-122e04e0b138\") " pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:45.712179 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.711329 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17696d35-744c-434c-94de-33ec719feac9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hq4sb\" (UID: \"17696d35-744c-434c-94de-33ec719feac9\") " pod="openshift-insights/insights-runtime-extractor-hq4sb" Oct 30 21:13:45.712179 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.711373 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-sysctl-d\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:45.712179 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.711415 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d4aae53-afc9-4373-953d-b9b7c323e3ee-host-slash\") pod \"iptables-alerter-jfprx\" (UID: \"1d4aae53-afc9-4373-953d-b9b7c323e3ee\") " pod="openshift-network-operator/iptables-alerter-jfprx" Oct 30 21:13:45.712179 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.711475 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtvrj\" (UniqueName: \"kubernetes.io/projected/9597db44-1ca0-4fa8-ade7-cb55eeca9593-kube-api-access-wtvrj\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:45.712179 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.711531 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4eed118d-1567-4be4-8c90-5dc0e7478827-metrics-tls\") pod \"dns-default-spbvd\" (UID: \"4eed118d-1567-4be4-8c90-5dc0e7478827\") " pod="openshift-dns/dns-default-spbvd" Oct 30 21:13:45.712179 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.711649 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-systemd\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:45.712179 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.711701 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-var-lib-kubelet\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:45.712179 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.711748 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54b8c0a5-e1a7-4d02-a402-b3cadc139e28-serviceca\") pod \"node-ca-q8g7b\" (UID: \"54b8c0a5-e1a7-4d02-a402-b3cadc139e28\") " pod="openshift-image-registry/node-ca-q8g7b" Oct 30 21:13:45.712179 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.711783 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9597db44-1ca0-4fa8-ade7-cb55eeca9593-registration-dir\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:45.712179 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.711832 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/05d96464-f946-4a83-88e5-ececfee30b4c-hosts-file\") pod \"node-resolver-p4c9m\" (UID: \"05d96464-f946-4a83-88e5-ececfee30b4c\") " pod="openshift-dns/node-resolver-p4c9m" Oct 30 21:13:45.712179 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.711862 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-host-var-lib-cni-multus\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.712179 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.711898 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-hostroot\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.712179 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.711927 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-multus-conf-dir\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.712179 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.711995 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/18d4542b-53c1-473f-9605-c82fd5a642f4-ovnkube-config\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.713243 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.712261 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/18d4542b-53c1-473f-9605-c82fd5a642f4-ovnkube-config\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.713243 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.712937 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54b8c0a5-e1a7-4d02-a402-b3cadc139e28-serviceca\") pod \"node-ca-q8g7b\" (UID: \"54b8c0a5-e1a7-4d02-a402-b3cadc139e28\") " pod="openshift-image-registry/node-ca-q8g7b" Oct 30 21:13:45.768299 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.768245 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvbqm\" (UniqueName: \"kubernetes.io/projected/25ca3b5e-7eba-4411-be09-5da0fe41e473-kube-api-access-mvbqm\") pod \"multus-additional-cni-plugins-brt6h\" (UID: \"25ca3b5e-7eba-4411-be09-5da0fe41e473\") " pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:45.768299 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.768245 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg66z\" (UniqueName: \"kubernetes.io/projected/54b8c0a5-e1a7-4d02-a402-b3cadc139e28-kube-api-access-jg66z\") pod \"node-ca-q8g7b\" (UID: \"54b8c0a5-e1a7-4d02-a402-b3cadc139e28\") " pod="openshift-image-registry/node-ca-q8g7b" Oct 30 21:13:45.768299 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.768293 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdp4m\" (UniqueName: \"kubernetes.io/projected/f96e4d00-99fb-4196-a1a9-83599bfd875f-kube-api-access-kdp4m\") pod \"service-ca-5fffc97bdf-zxlqv\" (UID: \"f96e4d00-99fb-4196-a1a9-83599bfd875f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zxlqv" Oct 30 21:13:45.771259 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.771238 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsrnm\" (UniqueName: \"kubernetes.io/projected/19538cda-3ba8-4bad-bb47-23a2511736e7-kube-api-access-bsrnm\") pod \"network-check-target-lw64d\" (UID: \"19538cda-3ba8-4bad-bb47-23a2511736e7\") " pod="openshift-network-diagnostics/network-check-target-lw64d" Oct 30 21:13:45.777243 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.777226 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfs8x\" (UniqueName: \"kubernetes.io/projected/70c3c6bc-c455-4fe6-9099-1ceb92abeaf0-kube-api-access-gfs8x\") pod \"migrator-66df5d496b-hz5dc\" (UID: \"70c3c6bc-c455-4fe6-9099-1ceb92abeaf0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hz5dc" Oct 30 21:13:45.777515 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.777500 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl5cb\" (UniqueName: \"kubernetes.io/projected/f285a424-0cc2-41b8-99b9-92db326ea01d-kube-api-access-gl5cb\") pod \"periodic-gathering-89mnx-pncws\" (UID: \"f285a424-0cc2-41b8-99b9-92db326ea01d\") " pod="openshift-insights/periodic-gathering-89mnx-pncws" Oct 30 21:13:45.777556 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.777504 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ghv7\" (UniqueName: \"kubernetes.io/projected/17696d35-744c-434c-94de-33ec719feac9-kube-api-access-5ghv7\") pod \"insights-runtime-extractor-hq4sb\" (UID: \"17696d35-744c-434c-94de-33ec719feac9\") " pod="openshift-insights/insights-runtime-extractor-hq4sb" Oct 30 21:13:45.778785 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.778765 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtvrj\" (UniqueName: \"kubernetes.io/projected/9597db44-1ca0-4fa8-ade7-cb55eeca9593-kube-api-access-wtvrj\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:45.779161 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.779146 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8snd\" (UniqueName: \"kubernetes.io/projected/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-kube-api-access-k8snd\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.800197 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.800172 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk4wl\" (UniqueName: \"kubernetes.io/projected/4eed118d-1567-4be4-8c90-5dc0e7478827-kube-api-access-tk4wl\") pod \"dns-default-spbvd\" (UID: \"4eed118d-1567-4be4-8c90-5dc0e7478827\") " pod="openshift-dns/dns-default-spbvd" Oct 30 21:13:45.800301 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.800281 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cctxz\" (UniqueName: \"kubernetes.io/projected/05d96464-f946-4a83-88e5-ececfee30b4c-kube-api-access-cctxz\") pod \"node-resolver-p4c9m\" (UID: \"05d96464-f946-4a83-88e5-ececfee30b4c\") " pod="openshift-dns/node-resolver-p4c9m" Oct 30 21:13:45.800389 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.800367 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8884\" (UniqueName: \"kubernetes.io/projected/30bde564-b785-4359-9377-ffec880fb18c-kube-api-access-l8884\") pod \"network-metrics-daemon-n4g68\" (UID: \"30bde564-b785-4359-9377-ffec880fb18c\") " pod="openshift-multus/network-metrics-daemon-n4g68" Oct 30 21:13:45.800485 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.800470 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl9th\" (UniqueName: \"kubernetes.io/projected/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-kube-api-access-hl9th\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:45.800817 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.800799 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k52xf\" (UniqueName: \"kubernetes.io/projected/18d4542b-53c1-473f-9605-c82fd5a642f4-kube-api-access-k52xf\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.800897 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.800873 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvgz2\" (UniqueName: \"kubernetes.io/projected/1d4aae53-afc9-4373-953d-b9b7c323e3ee-kube-api-access-tvgz2\") pod \"iptables-alerter-jfprx\" (UID: \"1d4aae53-afc9-4373-953d-b9b7c323e3ee\") " pod="openshift-network-operator/iptables-alerter-jfprx" Oct 30 21:13:45.800937 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.800904 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbrm8\" (UniqueName: \"kubernetes.io/projected/3d528ee4-1b51-4fdb-9166-56f14060d6af-kube-api-access-kbrm8\") pod \"ingress-canary-2kqn4\" (UID: \"3d528ee4-1b51-4fdb-9166-56f14060d6af\") " pod="openshift-ingress-canary/ingress-canary-2kqn4" Oct 30 21:13:45.812640 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.812622 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-multus-socket-dir-parent\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.812699 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.812646 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-run-openvswitch\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.812699 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.812661 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-run-ovn\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.812699 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.812676 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-cni-bin\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.812866 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.812692 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-run\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:45.812866 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.812721 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-run-openvswitch\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.812866 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.812729 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-run-netns\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.812866 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.812736 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-run-ovn\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.812866 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.812752 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/25ca3b5e-7eba-4411-be09-5da0fe41e473-cnibin\") pod \"multus-additional-cni-plugins-brt6h\" (UID: \"25ca3b5e-7eba-4411-be09-5da0fe41e473\") " pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:45.812866 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.812761 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-cni-bin\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.812866 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.812767 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-multus-socket-dir-parent\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.812866 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.812776 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9597db44-1ca0-4fa8-ade7-cb55eeca9593-device-dir\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:45.812866 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.812778 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-run\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:45.812866 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.812806 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-host-var-lib-kubelet\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.812866 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.812801 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-run-netns\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.812866 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.812812 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9597db44-1ca0-4fa8-ade7-cb55eeca9593-device-dir\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:45.812866 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.812804 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/25ca3b5e-7eba-4411-be09-5da0fe41e473-cnibin\") pod \"multus-additional-cni-plugins-brt6h\" (UID: \"25ca3b5e-7eba-4411-be09-5da0fe41e473\") " pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:45.812866 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.812844 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-host-var-lib-kubelet\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.812866 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.812859 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-slash\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.812877 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-var-lib-openvswitch\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.812897 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-host\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.812908 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-var-lib-openvswitch\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.812922 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-kubelet\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.812928 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-slash\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.812937 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3341bf47-8c8f-4073-8758-122e04e0b138-dbus\") pod \"global-pull-secret-syncer-jztdg\" (UID: \"3341bf47-8c8f-4073-8758-122e04e0b138\") " pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.812952 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-sysctl-d\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.812973 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-host\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.812986 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-kubelet\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.812985 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d4aae53-afc9-4373-953d-b9b7c323e3ee-host-slash\") pod \"iptables-alerter-jfprx\" (UID: \"1d4aae53-afc9-4373-953d-b9b7c323e3ee\") " pod="openshift-network-operator/iptables-alerter-jfprx" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813010 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d4aae53-afc9-4373-953d-b9b7c323e3ee-host-slash\") pod \"iptables-alerter-jfprx\" (UID: \"1d4aae53-afc9-4373-953d-b9b7c323e3ee\") " pod="openshift-network-operator/iptables-alerter-jfprx" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813023 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4eed118d-1567-4be4-8c90-5dc0e7478827-metrics-tls\") pod \"dns-default-spbvd\" (UID: \"4eed118d-1567-4be4-8c90-5dc0e7478827\") " pod="openshift-dns/dns-default-spbvd" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813039 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-systemd\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813054 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-var-lib-kubelet\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813072 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-sysctl-d\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813036 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3341bf47-8c8f-4073-8758-122e04e0b138-dbus\") pod \"global-pull-secret-syncer-jztdg\" (UID: \"3341bf47-8c8f-4073-8758-122e04e0b138\") " pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813082 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9597db44-1ca0-4fa8-ade7-cb55eeca9593-registration-dir\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813110 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/05d96464-f946-4a83-88e5-ececfee30b4c-hosts-file\") pod \"node-resolver-p4c9m\" (UID: \"05d96464-f946-4a83-88e5-ececfee30b4c\") " pod="openshift-dns/node-resolver-p4c9m" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813113 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-var-lib-kubelet\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813123 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9597db44-1ca0-4fa8-ade7-cb55eeca9593-registration-dir\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813134 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-systemd\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813134 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-host-var-lib-cni-multus\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813162 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/05d96464-f946-4a83-88e5-ececfee30b4c-hosts-file\") pod \"node-resolver-p4c9m\" (UID: \"05d96464-f946-4a83-88e5-ececfee30b4c\") " pod="openshift-dns/node-resolver-p4c9m" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813164 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-host-var-lib-cni-multus\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813175 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-hostroot\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813196 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-multus-conf-dir\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813213 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/25ca3b5e-7eba-4411-be09-5da0fe41e473-tuning-conf-dir\") pod \"multus-additional-cni-plugins-brt6h\" (UID: \"25ca3b5e-7eba-4411-be09-5da0fe41e473\") " pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813215 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-hostroot\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813246 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/25ca3b5e-7eba-4411-be09-5da0fe41e473-tuning-conf-dir\") pod \"multus-additional-cni-plugins-brt6h\" (UID: \"25ca3b5e-7eba-4411-be09-5da0fe41e473\") " pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813241 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-sysctl-conf\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813262 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-multus-conf-dir\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813275 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-host-run-k8s-cni-cncf-io\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813289 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-etc-kubernetes\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813302 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54b8c0a5-e1a7-4d02-a402-b3cadc139e28-host\") pod \"node-ca-q8g7b\" (UID: \"54b8c0a5-e1a7-4d02-a402-b3cadc139e28\") " pod="openshift-image-registry/node-ca-q8g7b" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813321 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9597db44-1ca0-4fa8-ade7-cb55eeca9593-etc-selinux\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813335 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-host-run-k8s-cni-cncf-io\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813363 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-etc-kubernetes\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813381 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-sysctl-conf\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813384 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54b8c0a5-e1a7-4d02-a402-b3cadc139e28-host\") pod \"node-ca-q8g7b\" (UID: \"54b8c0a5-e1a7-4d02-a402-b3cadc139e28\") " pod="openshift-image-registry/node-ca-q8g7b" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813361 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-host-run-netns\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813413 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-host-run-netns\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813418 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9597db44-1ca0-4fa8-ade7-cb55eeca9593-etc-selinux\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813416 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-cni-netd\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813441 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-cni-netd\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813420 4560 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813492 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/17696d35-744c-434c-94de-33ec719feac9-crio-socket\") pod \"insights-runtime-extractor-hq4sb\" (UID: \"17696d35-744c-434c-94de-33ec719feac9\") " pod="openshift-insights/insights-runtime-extractor-hq4sb" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813452 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/17696d35-744c-434c-94de-33ec719feac9-crio-socket\") pod \"insights-runtime-extractor-hq4sb\" (UID: \"17696d35-744c-434c-94de-33ec719feac9\") " pod="openshift-insights/insights-runtime-extractor-hq4sb" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813539 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9597db44-1ca0-4fa8-ade7-cb55eeca9593-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:45.813530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813562 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-log-socket\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813603 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-log-socket\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813606 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9597db44-1ca0-4fa8-ade7-cb55eeca9593-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813633 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9597db44-1ca0-4fa8-ade7-cb55eeca9593-sys-fs\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813660 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3341bf47-8c8f-4073-8758-122e04e0b138-kubelet-config\") pod \"global-pull-secret-syncer-jztdg\" (UID: \"3341bf47-8c8f-4073-8758-122e04e0b138\") " pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813685 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30bde564-b785-4359-9377-ffec880fb18c-metrics-certs\") pod \"network-metrics-daemon-n4g68\" (UID: \"30bde564-b785-4359-9377-ffec880fb18c\") " pod="openshift-multus/network-metrics-daemon-n4g68" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813705 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9597db44-1ca0-4fa8-ade7-cb55eeca9593-sys-fs\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813709 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-run-systemd\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813732 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-modprobe-d\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813740 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3341bf47-8c8f-4073-8758-122e04e0b138-kubelet-config\") pod \"global-pull-secret-syncer-jztdg\" (UID: \"3341bf47-8c8f-4073-8758-122e04e0b138\") " pod="kube-system/global-pull-secret-syncer-jztdg" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813748 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-sysconfig\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813756 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-run-systemd\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813776 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-lib-modules\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813789 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-sysconfig\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813801 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-cnibin\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813826 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-host-var-lib-cni-bin\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813837 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-modprobe-d\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813848 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-host-run-multus-certs\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813861 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-lib-modules\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813874 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-system-cni-dir\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813895 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-host-var-lib-cni-bin\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813903 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-system-cni-dir\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813903 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-host-run-multus-certs\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813878 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-cnibin\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813902 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-multus-cni-dir\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813947 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-systemd-units\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813953 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c877970-90d9-43ba-9c9b-4a22fbda1dc0-multus-cni-dir\") pod \"multus-zn2wp\" (UID: \"2c877970-90d9-43ba-9c9b-4a22fbda1dc0\") " pod="openshift-multus/multus-zn2wp" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813982 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-systemd-units\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.813987 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25ca3b5e-7eba-4411-be09-5da0fe41e473-system-cni-dir\") pod \"multus-additional-cni-plugins-brt6h\" (UID: \"25ca3b5e-7eba-4411-be09-5da0fe41e473\") " pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.814005 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25ca3b5e-7eba-4411-be09-5da0fe41e473-system-cni-dir\") pod \"multus-additional-cni-plugins-brt6h\" (UID: \"25ca3b5e-7eba-4411-be09-5da0fe41e473\") " pod="openshift-multus/multus-additional-cni-plugins-brt6h" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.814008 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-node-log\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.814030 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-run-ovn-kubernetes\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.814036 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-node-log\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.814051 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9597db44-1ca0-4fa8-ade7-cb55eeca9593-socket-dir\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.814073 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-run-ovn-kubernetes\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.814076 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-etc-openvswitch\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.814110 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.814120 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9597db44-1ca0-4fa8-ade7-cb55eeca9593-socket-dir\") pod \"aws-ebs-csi-driver-node-bd5g2\" (UID: \"9597db44-1ca0-4fa8-ade7-cb55eeca9593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd5g2" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.814126 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-etc-openvswitch\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.814147 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-kubernetes\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.814160 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18d4542b-53c1-473f-9605-c82fd5a642f4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jbjqt\" (UID: \"18d4542b-53c1-473f-9605-c82fd5a642f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.814173 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-sys\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.814201 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-etc-kubernetes\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:45.815070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.814215 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4e3a1e7-fed8-4969-b9ae-baec7c2dade8-sys\") pod \"tuned-pgnjv\" (UID: \"d4e3a1e7-fed8-4969-b9ae-baec7c2dade8\") " pod="openshift-cluster-node-tuning-operator/tuned-pgnjv" Oct 30 21:13:45.816206 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.815511 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4eed118d-1567-4be4-8c90-5dc0e7478827-metrics-tls\") pod \"dns-default-spbvd\" (UID: \"4eed118d-1567-4be4-8c90-5dc0e7478827\") " pod="openshift-dns/dns-default-spbvd" Oct 30 21:13:45.816206 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.815961 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30bde564-b785-4359-9377-ffec880fb18c-metrics-certs\") pod \"network-metrics-daemon-n4g68\" (UID: \"30bde564-b785-4359-9377-ffec880fb18c\") " pod="openshift-multus/network-metrics-daemon-n4g68" Oct 30 21:13:45.907461 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.907439 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4g68" Oct 30 21:13:45.910376 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.910357 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-spbvd" Oct 30 21:13:45.912196 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:45.912175 4560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:13:46.337210 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:46.337176 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n4g68"] Oct 30 21:13:46.345061 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:46.345030 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30bde564_b785_4359_9377_ffec880fb18c.slice/crio-d650b4a110411db9083b25239af0acaee44509b4dd154617c5b237c4270d38d1 WatchSource:0}: Error finding container d650b4a110411db9083b25239af0acaee44509b4dd154617c5b237c4270d38d1: Status 404 returned error can't find the container with id d650b4a110411db9083b25239af0acaee44509b4dd154617c5b237c4270d38d1 Oct 30 21:13:46.359206 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:46.359180 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-spbvd"] Oct 30 21:13:46.366635 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:46.366604 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eed118d_1567_4be4_8c90_5dc0e7478827.slice/crio-658346e11626b9da1cd9b11e9c04e03dd990cd687757699cd22f32991787f0a4 WatchSource:0}: Error finding container 658346e11626b9da1cd9b11e9c04e03dd990cd687757699cd22f32991787f0a4: Status 404 returned error can't find the container with id 658346e11626b9da1cd9b11e9c04e03dd990cd687757699cd22f32991787f0a4 Oct 30 21:13:46.394466 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:46.394445 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lw64d" Oct 30 21:13:46.616549 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:46.616470 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-spbvd" event={"ID":"4eed118d-1567-4be4-8c90-5dc0e7478827","Type":"ContainerStarted","Data":"658346e11626b9da1cd9b11e9c04e03dd990cd687757699cd22f32991787f0a4"} Oct 30 21:13:46.617653 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:46.617620 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n4g68" event={"ID":"30bde564-b785-4359-9377-ffec880fb18c","Type":"ContainerStarted","Data":"d650b4a110411db9083b25239af0acaee44509b4dd154617c5b237c4270d38d1"} Oct 30 21:13:47.624776 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:47.624703 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hq4sb" event={"ID":"17696d35-744c-434c-94de-33ec719feac9","Type":"ContainerStarted","Data":"7e058775c996b459486bdd7a1140938060f1e6134e64a7aa402e9d6bfcb4589d"} Oct 30 21:13:48.630361 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:48.630314 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n4g68" event={"ID":"30bde564-b785-4359-9377-ffec880fb18c","Type":"ContainerStarted","Data":"a024abd11bbcecef914aea9518736153697696537dd93f12582471310c9938b6"} Oct 30 21:13:48.630799 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:48.630399 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n4g68" event={"ID":"30bde564-b785-4359-9377-ffec880fb18c","Type":"ContainerStarted","Data":"e884f767bc2233532add26af4d6660b289db73c27f0edda8b9dc033862c8910a"} Oct 30 21:13:48.915158 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:48.915087 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:49.637848 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:49.637772 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-spbvd" event={"ID":"4eed118d-1567-4be4-8c90-5dc0e7478827","Type":"ContainerStarted","Data":"58da796c0d721ed8664ea003f6ed37af79a701861d2fbc023a0ed1c6274fa161"} Oct 30 21:13:49.637848 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:49.637822 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-spbvd" event={"ID":"4eed118d-1567-4be4-8c90-5dc0e7478827","Type":"ContainerStarted","Data":"efb55cab1c4057e3f99eea9937e9ef05e5a80a6844a2564f4b12727a6b987a3a"} Oct 30 21:13:49.638274 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:49.637881 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-spbvd" Oct 30 21:13:49.640032 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:49.640008 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hq4sb" event={"ID":"17696d35-744c-434c-94de-33ec719feac9","Type":"ContainerStarted","Data":"50ea4e2a13e035a859a8c3fb6296a7c4ffe80261ba7807961db20026f3cfb129"} Oct 30 21:13:49.682959 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:13:49.682927 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:13:55.215181 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:55.215150 4560 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-83.us-west-2.compute.internal" event="NodeReady" Oct 30 21:13:55.885062 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:55.885036 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:13:55.885268 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:55.885255 4560 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:13:57.162330 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:57.162304 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vpxpj" Oct 30 21:13:57.693373 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:57.693326 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-59f7d4445f-5448l"] Oct 30 21:13:57.719107 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:57.719084 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59f7d4445f-5448l" Oct 30 21:13:57.739787 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:57.739765 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59f7d4445f-5448l"] Oct 30 21:13:57.774585 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:57.774564 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 21:13:57.782785 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:57.782762 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b7efb2ae-833c-49e6-b863-b62e11fb20c9-bound-sa-token\") pod \"image-registry-59f7d4445f-5448l\" (UID: \"b7efb2ae-833c-49e6-b863-b62e11fb20c9\") " pod="openshift-image-registry/image-registry-59f7d4445f-5448l" Oct 30 21:13:57.782872 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:57.782790 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b7efb2ae-833c-49e6-b863-b62e11fb20c9-ca-trust-extracted\") pod \"image-registry-59f7d4445f-5448l\" (UID: \"b7efb2ae-833c-49e6-b863-b62e11fb20c9\") " pod="openshift-image-registry/image-registry-59f7d4445f-5448l" Oct 30 21:13:57.782872 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:57.782809 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b7efb2ae-833c-49e6-b863-b62e11fb20c9-registry-tls\") pod \"image-registry-59f7d4445f-5448l\" (UID: \"b7efb2ae-833c-49e6-b863-b62e11fb20c9\") " pod="openshift-image-registry/image-registry-59f7d4445f-5448l" Oct 30 21:13:57.782872 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:57.782825 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b7efb2ae-833c-49e6-b863-b62e11fb20c9-registry-certificates\") pod \"image-registry-59f7d4445f-5448l\" (UID: \"b7efb2ae-833c-49e6-b863-b62e11fb20c9\") " pod="openshift-image-registry/image-registry-59f7d4445f-5448l" Oct 30 21:13:57.782986 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:57.782876 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b7efb2ae-833c-49e6-b863-b62e11fb20c9-installation-pull-secrets\") pod \"image-registry-59f7d4445f-5448l\" (UID: \"b7efb2ae-833c-49e6-b863-b62e11fb20c9\") " pod="openshift-image-registry/image-registry-59f7d4445f-5448l" Oct 30 21:13:57.782986 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:57.782971 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b7efb2ae-833c-49e6-b863-b62e11fb20c9-image-registry-private-configuration\") pod \"image-registry-59f7d4445f-5448l\" (UID: \"b7efb2ae-833c-49e6-b863-b62e11fb20c9\") " pod="openshift-image-registry/image-registry-59f7d4445f-5448l" Oct 30 21:13:57.783052 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:57.782991 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b7efb2ae-833c-49e6-b863-b62e11fb20c9-trusted-ca\") pod \"image-registry-59f7d4445f-5448l\" (UID: \"b7efb2ae-833c-49e6-b863-b62e11fb20c9\") " pod="openshift-image-registry/image-registry-59f7d4445f-5448l" Oct 30 21:13:57.783052 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:57.783026 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz77x\" (UniqueName: \"kubernetes.io/projected/b7efb2ae-833c-49e6-b863-b62e11fb20c9-kube-api-access-zz77x\") pod \"image-registry-59f7d4445f-5448l\" (UID: \"b7efb2ae-833c-49e6-b863-b62e11fb20c9\") " pod="openshift-image-registry/image-registry-59f7d4445f-5448l" Oct 30 21:13:57.785864 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:57.785840 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lrfkn\"" Oct 30 21:13:57.785864 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:57.785856 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 21:13:57.785981 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:57.785933 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 21:13:57.868335 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:57.868313 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 21:13:57.883410 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:57.883385 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b7efb2ae-833c-49e6-b863-b62e11fb20c9-bound-sa-token\") pod \"image-registry-59f7d4445f-5448l\" (UID: \"b7efb2ae-833c-49e6-b863-b62e11fb20c9\") " pod="openshift-image-registry/image-registry-59f7d4445f-5448l" Oct 30 21:13:57.883488 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:57.883418 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b7efb2ae-833c-49e6-b863-b62e11fb20c9-ca-trust-extracted\") pod \"image-registry-59f7d4445f-5448l\" (UID: \"b7efb2ae-833c-49e6-b863-b62e11fb20c9\") " pod="openshift-image-registry/image-registry-59f7d4445f-5448l" Oct 30 21:13:57.883488 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:57.883436 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b7efb2ae-833c-49e6-b863-b62e11fb20c9-registry-tls\") pod \"image-registry-59f7d4445f-5448l\" (UID: \"b7efb2ae-833c-49e6-b863-b62e11fb20c9\") " pod="openshift-image-registry/image-registry-59f7d4445f-5448l" Oct 30 21:13:57.883488 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:57.883461 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b7efb2ae-833c-49e6-b863-b62e11fb20c9-registry-certificates\") pod \"image-registry-59f7d4445f-5448l\" (UID: \"b7efb2ae-833c-49e6-b863-b62e11fb20c9\") " pod="openshift-image-registry/image-registry-59f7d4445f-5448l" Oct 30 21:13:57.883628 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:57.883486 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b7efb2ae-833c-49e6-b863-b62e11fb20c9-installation-pull-secrets\") pod \"image-registry-59f7d4445f-5448l\" (UID: \"b7efb2ae-833c-49e6-b863-b62e11fb20c9\") " pod="openshift-image-registry/image-registry-59f7d4445f-5448l" Oct 30 21:13:57.883628 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:57.883515 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b7efb2ae-833c-49e6-b863-b62e11fb20c9-image-registry-private-configuration\") pod \"image-registry-59f7d4445f-5448l\" (UID: \"b7efb2ae-833c-49e6-b863-b62e11fb20c9\") " pod="openshift-image-registry/image-registry-59f7d4445f-5448l" Oct 30 21:13:57.883628 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:57.883538 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b7efb2ae-833c-49e6-b863-b62e11fb20c9-trusted-ca\") pod \"image-registry-59f7d4445f-5448l\" (UID: \"b7efb2ae-833c-49e6-b863-b62e11fb20c9\") " pod="openshift-image-registry/image-registry-59f7d4445f-5448l" Oct 30 21:13:57.883628 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:57.883578 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz77x\" (UniqueName: \"kubernetes.io/projected/b7efb2ae-833c-49e6-b863-b62e11fb20c9-kube-api-access-zz77x\") pod \"image-registry-59f7d4445f-5448l\" (UID: \"b7efb2ae-833c-49e6-b863-b62e11fb20c9\") " pod="openshift-image-registry/image-registry-59f7d4445f-5448l" Oct 30 21:13:57.883854 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:57.883833 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b7efb2ae-833c-49e6-b863-b62e11fb20c9-ca-trust-extracted\") pod \"image-registry-59f7d4445f-5448l\" (UID: \"b7efb2ae-833c-49e6-b863-b62e11fb20c9\") " pod="openshift-image-registry/image-registry-59f7d4445f-5448l" Oct 30 21:13:57.884312 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:57.884294 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b7efb2ae-833c-49e6-b863-b62e11fb20c9-registry-certificates\") pod \"image-registry-59f7d4445f-5448l\" (UID: \"b7efb2ae-833c-49e6-b863-b62e11fb20c9\") " pod="openshift-image-registry/image-registry-59f7d4445f-5448l" Oct 30 21:13:57.884609 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:57.884586 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b7efb2ae-833c-49e6-b863-b62e11fb20c9-trusted-ca\") pod \"image-registry-59f7d4445f-5448l\" (UID: \"b7efb2ae-833c-49e6-b863-b62e11fb20c9\") " pod="openshift-image-registry/image-registry-59f7d4445f-5448l" Oct 30 21:13:57.885900 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:57.885882 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b7efb2ae-833c-49e6-b863-b62e11fb20c9-installation-pull-secrets\") pod \"image-registry-59f7d4445f-5448l\" (UID: \"b7efb2ae-833c-49e6-b863-b62e11fb20c9\") " pod="openshift-image-registry/image-registry-59f7d4445f-5448l" Oct 30 21:13:57.886012 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:57.885997 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b7efb2ae-833c-49e6-b863-b62e11fb20c9-registry-tls\") pod \"image-registry-59f7d4445f-5448l\" (UID: \"b7efb2ae-833c-49e6-b863-b62e11fb20c9\") " pod="openshift-image-registry/image-registry-59f7d4445f-5448l" Oct 30 21:13:57.886012 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:57.886006 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b7efb2ae-833c-49e6-b863-b62e11fb20c9-image-registry-private-configuration\") pod \"image-registry-59f7d4445f-5448l\" (UID: \"b7efb2ae-833c-49e6-b863-b62e11fb20c9\") " pod="openshift-image-registry/image-registry-59f7d4445f-5448l" Oct 30 21:13:57.940717 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:57.940692 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz77x\" (UniqueName: \"kubernetes.io/projected/b7efb2ae-833c-49e6-b863-b62e11fb20c9-kube-api-access-zz77x\") pod \"image-registry-59f7d4445f-5448l\" (UID: \"b7efb2ae-833c-49e6-b863-b62e11fb20c9\") " pod="openshift-image-registry/image-registry-59f7d4445f-5448l" Oct 30 21:13:57.941161 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:57.941141 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b7efb2ae-833c-49e6-b863-b62e11fb20c9-bound-sa-token\") pod \"image-registry-59f7d4445f-5448l\" (UID: \"b7efb2ae-833c-49e6-b863-b62e11fb20c9\") " pod="openshift-image-registry/image-registry-59f7d4445f-5448l" Oct 30 21:13:58.035374 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:58.035354 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59f7d4445f-5448l" Oct 30 21:13:58.543275 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:58.543237 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59f7d4445f-5448l"] Oct 30 21:13:58.583248 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:13:58.583223 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7efb2ae_833c_49e6_b863_b62e11fb20c9.slice/crio-a8bbd7e129d3e9266d23c2e320482f465d30204f7b9a4e9061b95dd4ebf1b66a WatchSource:0}: Error finding container a8bbd7e129d3e9266d23c2e320482f465d30204f7b9a4e9061b95dd4ebf1b66a: Status 404 returned error can't find the container with id a8bbd7e129d3e9266d23c2e320482f465d30204f7b9a4e9061b95dd4ebf1b66a Oct 30 21:13:58.670230 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:58.670196 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59f7d4445f-5448l" event={"ID":"b7efb2ae-833c-49e6-b863-b62e11fb20c9","Type":"ContainerStarted","Data":"ce4ebf4433b3db6539bdfedd1574cbfda50552a6f88ab5c6a41f80a4d31d9b81"} Oct 30 21:13:58.670329 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:58.670264 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59f7d4445f-5448l" event={"ID":"b7efb2ae-833c-49e6-b863-b62e11fb20c9","Type":"ContainerStarted","Data":"a8bbd7e129d3e9266d23c2e320482f465d30204f7b9a4e9061b95dd4ebf1b66a"} Oct 30 21:13:59.646129 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:59.646102 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-spbvd" Oct 30 21:13:59.673722 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:59.673695 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-59f7d4445f-5448l" Oct 30 21:13:59.878687 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:13:59.878628 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-59f7d4445f-5448l" podStartSLOduration=2.878611581 podStartE2EDuration="2.878611581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:59.877302078 +0000 UTC m=+15.432001852" watchObservedRunningTime="2026-10-30 21:13:59.878611581 +0000 UTC m=+15.433311358" Oct 30 21:14:04.837460 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:04.837429 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jbjqt" Oct 30 21:14:12.758763 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:12.758723 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cwbdq"] Oct 30 21:14:12.791255 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:12.791223 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cwbdq" Oct 30 21:14:12.845626 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:12.845595 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:14:12.882504 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:12.882475 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/949d11d7-1fa3-4c6f-b08c-faa10f082e00-node-exporter-textfile\") pod \"node-exporter-cwbdq\" (UID: \"949d11d7-1fa3-4c6f-b08c-faa10f082e00\") " pod="openshift-monitoring/node-exporter-cwbdq" Oct 30 21:14:12.882631 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:12.882509 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/949d11d7-1fa3-4c6f-b08c-faa10f082e00-node-exporter-accelerators-collector-config\") pod \"node-exporter-cwbdq\" (UID: \"949d11d7-1fa3-4c6f-b08c-faa10f082e00\") " pod="openshift-monitoring/node-exporter-cwbdq" Oct 30 21:14:12.882631 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:12.882545 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/949d11d7-1fa3-4c6f-b08c-faa10f082e00-metrics-client-ca\") pod \"node-exporter-cwbdq\" (UID: \"949d11d7-1fa3-4c6f-b08c-faa10f082e00\") " pod="openshift-monitoring/node-exporter-cwbdq" Oct 30 21:14:12.882631 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:12.882625 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/949d11d7-1fa3-4c6f-b08c-faa10f082e00-node-exporter-wtmp\") pod \"node-exporter-cwbdq\" (UID: \"949d11d7-1fa3-4c6f-b08c-faa10f082e00\") " pod="openshift-monitoring/node-exporter-cwbdq" Oct 30 21:14:12.882736 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:12.882653 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/949d11d7-1fa3-4c6f-b08c-faa10f082e00-sys\") pod \"node-exporter-cwbdq\" (UID: \"949d11d7-1fa3-4c6f-b08c-faa10f082e00\") " pod="openshift-monitoring/node-exporter-cwbdq" Oct 30 21:14:12.882736 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:12.882672 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/949d11d7-1fa3-4c6f-b08c-faa10f082e00-node-exporter-tls\") pod \"node-exporter-cwbdq\" (UID: \"949d11d7-1fa3-4c6f-b08c-faa10f082e00\") " pod="openshift-monitoring/node-exporter-cwbdq" Oct 30 21:14:12.882736 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:12.882694 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/949d11d7-1fa3-4c6f-b08c-faa10f082e00-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cwbdq\" (UID: \"949d11d7-1fa3-4c6f-b08c-faa10f082e00\") " pod="openshift-monitoring/node-exporter-cwbdq" Oct 30 21:14:12.882822 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:12.882765 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbsn4\" (UniqueName: \"kubernetes.io/projected/949d11d7-1fa3-4c6f-b08c-faa10f082e00-kube-api-access-xbsn4\") pod \"node-exporter-cwbdq\" (UID: \"949d11d7-1fa3-4c6f-b08c-faa10f082e00\") " pod="openshift-monitoring/node-exporter-cwbdq" Oct 30 21:14:12.882822 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:12.882797 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/949d11d7-1fa3-4c6f-b08c-faa10f082e00-root\") pod \"node-exporter-cwbdq\" (UID: \"949d11d7-1fa3-4c6f-b08c-faa10f082e00\") " pod="openshift-monitoring/node-exporter-cwbdq" Oct 30 21:14:12.891546 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:12.891527 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 21:14:12.891546 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:12.891531 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:14:12.891670 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:12.891615 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 21:14:12.938298 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:12.938274 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 21:14:12.938444 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:12.938405 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 21:14:12.938444 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:12.938419 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xhltr\"" Oct 30 21:14:12.984154 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:12.984117 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/949d11d7-1fa3-4c6f-b08c-faa10f082e00-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cwbdq\" (UID: \"949d11d7-1fa3-4c6f-b08c-faa10f082e00\") " pod="openshift-monitoring/node-exporter-cwbdq" Oct 30 21:14:12.984349 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:12.984166 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbsn4\" (UniqueName: \"kubernetes.io/projected/949d11d7-1fa3-4c6f-b08c-faa10f082e00-kube-api-access-xbsn4\") pod \"node-exporter-cwbdq\" (UID: \"949d11d7-1fa3-4c6f-b08c-faa10f082e00\") " pod="openshift-monitoring/node-exporter-cwbdq" Oct 30 21:14:12.984349 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:12.984192 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/949d11d7-1fa3-4c6f-b08c-faa10f082e00-root\") pod \"node-exporter-cwbdq\" (UID: \"949d11d7-1fa3-4c6f-b08c-faa10f082e00\") " pod="openshift-monitoring/node-exporter-cwbdq" Oct 30 21:14:12.984349 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:12.984228 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/949d11d7-1fa3-4c6f-b08c-faa10f082e00-node-exporter-textfile\") pod \"node-exporter-cwbdq\" (UID: \"949d11d7-1fa3-4c6f-b08c-faa10f082e00\") " pod="openshift-monitoring/node-exporter-cwbdq" Oct 30 21:14:12.984349 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:12.984254 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/949d11d7-1fa3-4c6f-b08c-faa10f082e00-node-exporter-accelerators-collector-config\") pod \"node-exporter-cwbdq\" (UID: \"949d11d7-1fa3-4c6f-b08c-faa10f082e00\") " pod="openshift-monitoring/node-exporter-cwbdq" Oct 30 21:14:12.984349 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:12.984293 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/949d11d7-1fa3-4c6f-b08c-faa10f082e00-metrics-client-ca\") pod \"node-exporter-cwbdq\" (UID: \"949d11d7-1fa3-4c6f-b08c-faa10f082e00\") " pod="openshift-monitoring/node-exporter-cwbdq" Oct 30 21:14:12.984349 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:12.984321 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/949d11d7-1fa3-4c6f-b08c-faa10f082e00-node-exporter-wtmp\") pod \"node-exporter-cwbdq\" (UID: \"949d11d7-1fa3-4c6f-b08c-faa10f082e00\") " pod="openshift-monitoring/node-exporter-cwbdq" Oct 30 21:14:12.984653 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:12.984364 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/949d11d7-1fa3-4c6f-b08c-faa10f082e00-sys\") pod \"node-exporter-cwbdq\" (UID: \"949d11d7-1fa3-4c6f-b08c-faa10f082e00\") " pod="openshift-monitoring/node-exporter-cwbdq" Oct 30 21:14:12.984653 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:12.984392 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/949d11d7-1fa3-4c6f-b08c-faa10f082e00-node-exporter-tls\") pod \"node-exporter-cwbdq\" (UID: \"949d11d7-1fa3-4c6f-b08c-faa10f082e00\") " pod="openshift-monitoring/node-exporter-cwbdq" Oct 30 21:14:12.984653 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:12.984410 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/949d11d7-1fa3-4c6f-b08c-faa10f082e00-sys\") pod \"node-exporter-cwbdq\" (UID: \"949d11d7-1fa3-4c6f-b08c-faa10f082e00\") " pod="openshift-monitoring/node-exporter-cwbdq" Oct 30 21:14:12.984653 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:12.984316 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/949d11d7-1fa3-4c6f-b08c-faa10f082e00-root\") pod \"node-exporter-cwbdq\" (UID: \"949d11d7-1fa3-4c6f-b08c-faa10f082e00\") " pod="openshift-monitoring/node-exporter-cwbdq" Oct 30 21:14:12.984653 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:12.984579 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/949d11d7-1fa3-4c6f-b08c-faa10f082e00-node-exporter-wtmp\") pod \"node-exporter-cwbdq\" (UID: \"949d11d7-1fa3-4c6f-b08c-faa10f082e00\") " pod="openshift-monitoring/node-exporter-cwbdq" Oct 30 21:14:12.984860 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:12.984757 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/949d11d7-1fa3-4c6f-b08c-faa10f082e00-node-exporter-textfile\") pod \"node-exporter-cwbdq\" (UID: \"949d11d7-1fa3-4c6f-b08c-faa10f082e00\") " pod="openshift-monitoring/node-exporter-cwbdq" Oct 30 21:14:12.984997 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:12.984970 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/949d11d7-1fa3-4c6f-b08c-faa10f082e00-node-exporter-accelerators-collector-config\") pod \"node-exporter-cwbdq\" (UID: \"949d11d7-1fa3-4c6f-b08c-faa10f082e00\") " pod="openshift-monitoring/node-exporter-cwbdq" Oct 30 21:14:12.985162 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:12.985144 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/949d11d7-1fa3-4c6f-b08c-faa10f082e00-metrics-client-ca\") pod \"node-exporter-cwbdq\" (UID: \"949d11d7-1fa3-4c6f-b08c-faa10f082e00\") " pod="openshift-monitoring/node-exporter-cwbdq" Oct 30 21:14:12.986539 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:12.986516 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/949d11d7-1fa3-4c6f-b08c-faa10f082e00-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cwbdq\" (UID: \"949d11d7-1fa3-4c6f-b08c-faa10f082e00\") " pod="openshift-monitoring/node-exporter-cwbdq" Oct 30 21:14:12.986628 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:12.986574 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/949d11d7-1fa3-4c6f-b08c-faa10f082e00-node-exporter-tls\") pod \"node-exporter-cwbdq\" (UID: \"949d11d7-1fa3-4c6f-b08c-faa10f082e00\") " pod="openshift-monitoring/node-exporter-cwbdq" Oct 30 21:14:13.042756 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:13.042659 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbsn4\" (UniqueName: \"kubernetes.io/projected/949d11d7-1fa3-4c6f-b08c-faa10f082e00-kube-api-access-xbsn4\") pod \"node-exporter-cwbdq\" (UID: \"949d11d7-1fa3-4c6f-b08c-faa10f082e00\") " pod="openshift-monitoring/node-exporter-cwbdq" Oct 30 21:14:13.140202 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:13.140164 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cwbdq" Oct 30 21:14:13.182645 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:14:13.182619 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod949d11d7_1fa3_4c6f_b08c_faa10f082e00.slice/crio-6fc03e5afeb48f14e411c28650b42066557e48dd0e1c47aa9482e2a186de54e7 WatchSource:0}: Error finding container 6fc03e5afeb48f14e411c28650b42066557e48dd0e1c47aa9482e2a186de54e7: Status 404 returned error can't find the container with id 6fc03e5afeb48f14e411c28650b42066557e48dd0e1c47aa9482e2a186de54e7 Oct 30 21:14:13.717477 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:13.717449 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cwbdq" event={"ID":"949d11d7-1fa3-4c6f-b08c-faa10f082e00","Type":"ContainerStarted","Data":"6fc03e5afeb48f14e411c28650b42066557e48dd0e1c47aa9482e2a186de54e7"} Oct 30 21:14:15.724995 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:15.724960 4560 generic.go:356] "Generic (PLEG): container finished" podID="949d11d7-1fa3-4c6f-b08c-faa10f082e00" containerID="99b42483aa496c5d328c7b3b888003165aa70b1dd0216ab77da7888eaa462940" exitCode=0 Oct 30 21:14:15.725400 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:15.725001 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cwbdq" event={"ID":"949d11d7-1fa3-4c6f-b08c-faa10f082e00","Type":"ContainerDied","Data":"99b42483aa496c5d328c7b3b888003165aa70b1dd0216ab77da7888eaa462940"} Oct 30 21:14:16.620207 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:16.620175 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lw64d" Oct 30 21:14:16.730431 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:16.730394 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cwbdq" event={"ID":"949d11d7-1fa3-4c6f-b08c-faa10f082e00","Type":"ContainerStarted","Data":"de73085e24789e2b34d90a6c82e39e9a365f421f1864bbeef68674948ec6d2ba"} Oct 30 21:14:16.730794 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:16.730441 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cwbdq" event={"ID":"949d11d7-1fa3-4c6f-b08c-faa10f082e00","Type":"ContainerStarted","Data":"577375fbbf572b288c7b568420d70c549a66264f42ec9089c209ffc6bf8e81a2"} Oct 30 21:14:16.860251 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:16.860198 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cwbdq" podStartSLOduration=2.77536123 podStartE2EDuration="4.860183365s" totalImagesPullingTime="1.999253093s" totalInitContainerRuntime="85.569042ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:14:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:16.859510166 +0000 UTC m=+32.414209943" watchObservedRunningTime="2026-10-30 21:14:16.860183365 +0000 UTC m=+32.414883120" Oct 30 21:14:17.578162 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:17.578134 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-756b9cccf4-cz4sx"] Oct 30 21:14:17.603655 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:17.603633 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-756b9cccf4-cz4sx" Oct 30 21:14:17.641688 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:17.641655 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-756b9cccf4-cz4sx"] Oct 30 21:14:17.656209 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:17.656180 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 21:14:17.656209 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:17.656210 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 21:14:17.687708 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:17.687688 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 21:14:17.687708 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:17.687694 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-q5qmm\"" Oct 30 21:14:17.701081 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:17.701059 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6q1b9uqo6cndv\"" Oct 30 21:14:17.701170 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:17.701133 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 21:14:17.717885 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:17.717862 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgtb9\" (UniqueName: \"kubernetes.io/projected/017ad089-d65d-47a3-9b1e-10fbb9290e19-kube-api-access-dgtb9\") pod \"metrics-server-756b9cccf4-cz4sx\" (UID: \"017ad089-d65d-47a3-9b1e-10fbb9290e19\") " pod="openshift-monitoring/metrics-server-756b9cccf4-cz4sx" Oct 30 21:14:17.718145 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:17.717905 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/017ad089-d65d-47a3-9b1e-10fbb9290e19-metrics-server-audit-profiles\") pod \"metrics-server-756b9cccf4-cz4sx\" (UID: \"017ad089-d65d-47a3-9b1e-10fbb9290e19\") " pod="openshift-monitoring/metrics-server-756b9cccf4-cz4sx" Oct 30 21:14:17.718145 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:17.717939 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/017ad089-d65d-47a3-9b1e-10fbb9290e19-secret-metrics-server-tls\") pod \"metrics-server-756b9cccf4-cz4sx\" (UID: \"017ad089-d65d-47a3-9b1e-10fbb9290e19\") " pod="openshift-monitoring/metrics-server-756b9cccf4-cz4sx" Oct 30 21:14:17.718145 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:17.717978 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/017ad089-d65d-47a3-9b1e-10fbb9290e19-client-ca-bundle\") pod \"metrics-server-756b9cccf4-cz4sx\" (UID: \"017ad089-d65d-47a3-9b1e-10fbb9290e19\") " pod="openshift-monitoring/metrics-server-756b9cccf4-cz4sx" Oct 30 21:14:17.718145 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:17.718099 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/017ad089-d65d-47a3-9b1e-10fbb9290e19-secret-metrics-server-client-certs\") pod \"metrics-server-756b9cccf4-cz4sx\" (UID: \"017ad089-d65d-47a3-9b1e-10fbb9290e19\") " pod="openshift-monitoring/metrics-server-756b9cccf4-cz4sx" Oct 30 21:14:17.718145 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:17.718132 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/017ad089-d65d-47a3-9b1e-10fbb9290e19-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-756b9cccf4-cz4sx\" (UID: \"017ad089-d65d-47a3-9b1e-10fbb9290e19\") " pod="openshift-monitoring/metrics-server-756b9cccf4-cz4sx" Oct 30 21:14:17.718328 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:17.718159 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/017ad089-d65d-47a3-9b1e-10fbb9290e19-audit-log\") pod \"metrics-server-756b9cccf4-cz4sx\" (UID: \"017ad089-d65d-47a3-9b1e-10fbb9290e19\") " pod="openshift-monitoring/metrics-server-756b9cccf4-cz4sx" Oct 30 21:14:17.818571 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:17.818545 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/017ad089-d65d-47a3-9b1e-10fbb9290e19-secret-metrics-server-client-certs\") pod \"metrics-server-756b9cccf4-cz4sx\" (UID: \"017ad089-d65d-47a3-9b1e-10fbb9290e19\") " pod="openshift-monitoring/metrics-server-756b9cccf4-cz4sx" Oct 30 21:14:17.818992 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:17.818581 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/017ad089-d65d-47a3-9b1e-10fbb9290e19-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-756b9cccf4-cz4sx\" (UID: \"017ad089-d65d-47a3-9b1e-10fbb9290e19\") " pod="openshift-monitoring/metrics-server-756b9cccf4-cz4sx" Oct 30 21:14:17.818992 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:17.818602 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/017ad089-d65d-47a3-9b1e-10fbb9290e19-audit-log\") pod \"metrics-server-756b9cccf4-cz4sx\" (UID: \"017ad089-d65d-47a3-9b1e-10fbb9290e19\") " pod="openshift-monitoring/metrics-server-756b9cccf4-cz4sx" Oct 30 21:14:17.818992 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:17.818751 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgtb9\" (UniqueName: \"kubernetes.io/projected/017ad089-d65d-47a3-9b1e-10fbb9290e19-kube-api-access-dgtb9\") pod \"metrics-server-756b9cccf4-cz4sx\" (UID: \"017ad089-d65d-47a3-9b1e-10fbb9290e19\") " pod="openshift-monitoring/metrics-server-756b9cccf4-cz4sx" Oct 30 21:14:17.818992 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:17.818813 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/017ad089-d65d-47a3-9b1e-10fbb9290e19-metrics-server-audit-profiles\") pod \"metrics-server-756b9cccf4-cz4sx\" (UID: \"017ad089-d65d-47a3-9b1e-10fbb9290e19\") " pod="openshift-monitoring/metrics-server-756b9cccf4-cz4sx" Oct 30 21:14:17.818992 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:17.818845 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/017ad089-d65d-47a3-9b1e-10fbb9290e19-secret-metrics-server-tls\") pod \"metrics-server-756b9cccf4-cz4sx\" (UID: \"017ad089-d65d-47a3-9b1e-10fbb9290e19\") " pod="openshift-monitoring/metrics-server-756b9cccf4-cz4sx" Oct 30 21:14:17.818992 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:17.818868 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/017ad089-d65d-47a3-9b1e-10fbb9290e19-client-ca-bundle\") pod \"metrics-server-756b9cccf4-cz4sx\" (UID: \"017ad089-d65d-47a3-9b1e-10fbb9290e19\") " pod="openshift-monitoring/metrics-server-756b9cccf4-cz4sx" Oct 30 21:14:17.819392 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:17.819369 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/017ad089-d65d-47a3-9b1e-10fbb9290e19-audit-log\") pod \"metrics-server-756b9cccf4-cz4sx\" (UID: \"017ad089-d65d-47a3-9b1e-10fbb9290e19\") " pod="openshift-monitoring/metrics-server-756b9cccf4-cz4sx" Oct 30 21:14:17.819575 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:17.819450 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/017ad089-d65d-47a3-9b1e-10fbb9290e19-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-756b9cccf4-cz4sx\" (UID: \"017ad089-d65d-47a3-9b1e-10fbb9290e19\") " pod="openshift-monitoring/metrics-server-756b9cccf4-cz4sx" Oct 30 21:14:17.820087 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:17.820062 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/017ad089-d65d-47a3-9b1e-10fbb9290e19-metrics-server-audit-profiles\") pod \"metrics-server-756b9cccf4-cz4sx\" (UID: \"017ad089-d65d-47a3-9b1e-10fbb9290e19\") " pod="openshift-monitoring/metrics-server-756b9cccf4-cz4sx" Oct 30 21:14:17.821220 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:17.821199 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/017ad089-d65d-47a3-9b1e-10fbb9290e19-secret-metrics-server-client-certs\") pod \"metrics-server-756b9cccf4-cz4sx\" (UID: \"017ad089-d65d-47a3-9b1e-10fbb9290e19\") " pod="openshift-monitoring/metrics-server-756b9cccf4-cz4sx" Oct 30 21:14:17.821533 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:17.821512 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/017ad089-d65d-47a3-9b1e-10fbb9290e19-client-ca-bundle\") pod \"metrics-server-756b9cccf4-cz4sx\" (UID: \"017ad089-d65d-47a3-9b1e-10fbb9290e19\") " pod="openshift-monitoring/metrics-server-756b9cccf4-cz4sx" Oct 30 21:14:17.821594 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:17.821578 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/017ad089-d65d-47a3-9b1e-10fbb9290e19-secret-metrics-server-tls\") pod \"metrics-server-756b9cccf4-cz4sx\" (UID: \"017ad089-d65d-47a3-9b1e-10fbb9290e19\") " pod="openshift-monitoring/metrics-server-756b9cccf4-cz4sx" Oct 30 21:14:17.876613 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:17.876559 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgtb9\" (UniqueName: \"kubernetes.io/projected/017ad089-d65d-47a3-9b1e-10fbb9290e19-kube-api-access-dgtb9\") pod \"metrics-server-756b9cccf4-cz4sx\" (UID: \"017ad089-d65d-47a3-9b1e-10fbb9290e19\") " pod="openshift-monitoring/metrics-server-756b9cccf4-cz4sx" Oct 30 21:14:17.928626 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:17.928601 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-756b9cccf4-cz4sx" Oct 30 21:14:18.377254 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:18.377221 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-756b9cccf4-cz4sx"] Oct 30 21:14:18.385366 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:14:18.385321 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod017ad089_d65d_47a3_9b1e_10fbb9290e19.slice/crio-e45c43907dfc433482a3ab38690f6a6eab1ed11aedbcd523484cbef995cd8538 WatchSource:0}: Error finding container e45c43907dfc433482a3ab38690f6a6eab1ed11aedbcd523484cbef995cd8538: Status 404 returned error can't find the container with id e45c43907dfc433482a3ab38690f6a6eab1ed11aedbcd523484cbef995cd8538 Oct 30 21:14:18.737677 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:18.737638 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-756b9cccf4-cz4sx" event={"ID":"017ad089-d65d-47a3-9b1e-10fbb9290e19","Type":"ContainerStarted","Data":"e45c43907dfc433482a3ab38690f6a6eab1ed11aedbcd523484cbef995cd8538"} Oct 30 21:14:19.688997 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:14:19.688956 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:14:20.680602 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:20.680536 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-59f7d4445f-5448l" Oct 30 21:14:20.746712 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:20.746680 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-756b9cccf4-cz4sx" event={"ID":"017ad089-d65d-47a3-9b1e-10fbb9290e19","Type":"ContainerStarted","Data":"40c69859fa05601b9928414ce76d988db675cbb4609f5eeb8554dc0132334cef"} Oct 30 21:14:20.965675 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:20.965584 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-756b9cccf4-cz4sx" podStartSLOduration=1.948885485 podStartE2EDuration="3.965570632s" totalImagesPullingTime="2.016685147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:20.965044222 +0000 UTC m=+36.519744000" watchObservedRunningTime="2026-10-30 21:14:20.965570632 +0000 UTC m=+36.520270410" Oct 30 21:14:26.587101 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:26.587066 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c89668d74-8fx8h"] Oct 30 21:14:26.589355 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:26.589319 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c89668d74-8fx8h" Oct 30 21:14:26.683227 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:26.683196 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85d7dd2c-56fb-404b-9e24-42413e0b18c9-tmp\") pod \"console-5c89668d74-8fx8h\" (UID: \"85d7dd2c-56fb-404b-9e24-42413e0b18c9\") " pod="openshift-console/console-5c89668d74-8fx8h" Oct 30 21:14:26.683227 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:26.683227 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85d7dd2c-56fb-404b-9e24-42413e0b18c9-oauth-serving-cert\") pod \"console-5c89668d74-8fx8h\" (UID: \"85d7dd2c-56fb-404b-9e24-42413e0b18c9\") " pod="openshift-console/console-5c89668d74-8fx8h" Oct 30 21:14:26.683421 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:26.683251 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85d7dd2c-56fb-404b-9e24-42413e0b18c9-console-oauth-config\") pod \"console-5c89668d74-8fx8h\" (UID: \"85d7dd2c-56fb-404b-9e24-42413e0b18c9\") " pod="openshift-console/console-5c89668d74-8fx8h" Oct 30 21:14:26.683421 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:26.683314 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgndk\" (UniqueName: \"kubernetes.io/projected/85d7dd2c-56fb-404b-9e24-42413e0b18c9-kube-api-access-hgndk\") pod \"console-5c89668d74-8fx8h\" (UID: \"85d7dd2c-56fb-404b-9e24-42413e0b18c9\") " pod="openshift-console/console-5c89668d74-8fx8h" Oct 30 21:14:26.683421 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:26.683417 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85d7dd2c-56fb-404b-9e24-42413e0b18c9-service-ca\") pod \"console-5c89668d74-8fx8h\" (UID: \"85d7dd2c-56fb-404b-9e24-42413e0b18c9\") " pod="openshift-console/console-5c89668d74-8fx8h" Oct 30 21:14:26.683516 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:26.683448 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85d7dd2c-56fb-404b-9e24-42413e0b18c9-console-config\") pod \"console-5c89668d74-8fx8h\" (UID: \"85d7dd2c-56fb-404b-9e24-42413e0b18c9\") " pod="openshift-console/console-5c89668d74-8fx8h" Oct 30 21:14:26.683516 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:26.683471 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85d7dd2c-56fb-404b-9e24-42413e0b18c9-console-serving-cert\") pod \"console-5c89668d74-8fx8h\" (UID: \"85d7dd2c-56fb-404b-9e24-42413e0b18c9\") " pod="openshift-console/console-5c89668d74-8fx8h" Oct 30 21:14:26.686119 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:26.686097 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 21:14:26.686273 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:26.686097 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 21:14:26.686398 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:26.686098 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 21:14:26.686398 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:26.686097 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 21:14:26.686398 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:26.686105 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 21:14:26.686398 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:26.686129 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 21:14:26.686398 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:26.686151 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 21:14:26.686398 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:26.686255 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bwl78\"" Oct 30 21:14:26.687002 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:26.686985 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c89668d74-8fx8h"] Oct 30 21:14:26.784370 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:26.784322 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85d7dd2c-56fb-404b-9e24-42413e0b18c9-service-ca\") pod \"console-5c89668d74-8fx8h\" (UID: \"85d7dd2c-56fb-404b-9e24-42413e0b18c9\") " pod="openshift-console/console-5c89668d74-8fx8h" Oct 30 21:14:26.784542 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:26.784380 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85d7dd2c-56fb-404b-9e24-42413e0b18c9-console-config\") pod \"console-5c89668d74-8fx8h\" (UID: \"85d7dd2c-56fb-404b-9e24-42413e0b18c9\") " pod="openshift-console/console-5c89668d74-8fx8h" Oct 30 21:14:26.784542 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:26.784403 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85d7dd2c-56fb-404b-9e24-42413e0b18c9-console-serving-cert\") pod \"console-5c89668d74-8fx8h\" (UID: \"85d7dd2c-56fb-404b-9e24-42413e0b18c9\") " pod="openshift-console/console-5c89668d74-8fx8h" Oct 30 21:14:26.784542 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:26.784430 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85d7dd2c-56fb-404b-9e24-42413e0b18c9-tmp\") pod \"console-5c89668d74-8fx8h\" (UID: \"85d7dd2c-56fb-404b-9e24-42413e0b18c9\") " pod="openshift-console/console-5c89668d74-8fx8h" Oct 30 21:14:26.784542 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:26.784444 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85d7dd2c-56fb-404b-9e24-42413e0b18c9-oauth-serving-cert\") pod \"console-5c89668d74-8fx8h\" (UID: \"85d7dd2c-56fb-404b-9e24-42413e0b18c9\") " pod="openshift-console/console-5c89668d74-8fx8h" Oct 30 21:14:26.784542 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:26.784468 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85d7dd2c-56fb-404b-9e24-42413e0b18c9-console-oauth-config\") pod \"console-5c89668d74-8fx8h\" (UID: \"85d7dd2c-56fb-404b-9e24-42413e0b18c9\") " pod="openshift-console/console-5c89668d74-8fx8h" Oct 30 21:14:26.784542 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:26.784494 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgndk\" (UniqueName: \"kubernetes.io/projected/85d7dd2c-56fb-404b-9e24-42413e0b18c9-kube-api-access-hgndk\") pod \"console-5c89668d74-8fx8h\" (UID: \"85d7dd2c-56fb-404b-9e24-42413e0b18c9\") " pod="openshift-console/console-5c89668d74-8fx8h" Oct 30 21:14:26.784953 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:26.784928 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85d7dd2c-56fb-404b-9e24-42413e0b18c9-tmp\") pod \"console-5c89668d74-8fx8h\" (UID: \"85d7dd2c-56fb-404b-9e24-42413e0b18c9\") " pod="openshift-console/console-5c89668d74-8fx8h" Oct 30 21:14:26.785363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:26.785186 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85d7dd2c-56fb-404b-9e24-42413e0b18c9-service-ca\") pod \"console-5c89668d74-8fx8h\" (UID: \"85d7dd2c-56fb-404b-9e24-42413e0b18c9\") " pod="openshift-console/console-5c89668d74-8fx8h" Oct 30 21:14:26.785363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:26.785201 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85d7dd2c-56fb-404b-9e24-42413e0b18c9-console-config\") pod \"console-5c89668d74-8fx8h\" (UID: \"85d7dd2c-56fb-404b-9e24-42413e0b18c9\") " pod="openshift-console/console-5c89668d74-8fx8h" Oct 30 21:14:26.785492 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:26.785411 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85d7dd2c-56fb-404b-9e24-42413e0b18c9-oauth-serving-cert\") pod \"console-5c89668d74-8fx8h\" (UID: \"85d7dd2c-56fb-404b-9e24-42413e0b18c9\") " pod="openshift-console/console-5c89668d74-8fx8h" Oct 30 21:14:26.786888 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:26.786866 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85d7dd2c-56fb-404b-9e24-42413e0b18c9-console-serving-cert\") pod \"console-5c89668d74-8fx8h\" (UID: \"85d7dd2c-56fb-404b-9e24-42413e0b18c9\") " pod="openshift-console/console-5c89668d74-8fx8h" Oct 30 21:14:26.786988 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:26.786972 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85d7dd2c-56fb-404b-9e24-42413e0b18c9-console-oauth-config\") pod \"console-5c89668d74-8fx8h\" (UID: \"85d7dd2c-56fb-404b-9e24-42413e0b18c9\") " pod="openshift-console/console-5c89668d74-8fx8h" Oct 30 21:14:26.842481 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:26.842421 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgndk\" (UniqueName: \"kubernetes.io/projected/85d7dd2c-56fb-404b-9e24-42413e0b18c9-kube-api-access-hgndk\") pod \"console-5c89668d74-8fx8h\" (UID: \"85d7dd2c-56fb-404b-9e24-42413e0b18c9\") " pod="openshift-console/console-5c89668d74-8fx8h" Oct 30 21:14:26.899453 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:26.899422 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c89668d74-8fx8h" Oct 30 21:14:27.270276 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:27.270245 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c89668d74-8fx8h"] Oct 30 21:14:27.277471 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:14:27.277444 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d7dd2c_56fb_404b_9e24_42413e0b18c9.slice/crio-3c8ea08172cbbeb7004140314b3ddadf268fbc5943d549c948a26a2fe000a7c4 WatchSource:0}: Error finding container 3c8ea08172cbbeb7004140314b3ddadf268fbc5943d549c948a26a2fe000a7c4: Status 404 returned error can't find the container with id 3c8ea08172cbbeb7004140314b3ddadf268fbc5943d549c948a26a2fe000a7c4 Oct 30 21:14:27.769161 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:27.769123 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c89668d74-8fx8h" event={"ID":"85d7dd2c-56fb-404b-9e24-42413e0b18c9","Type":"ContainerStarted","Data":"3c8ea08172cbbeb7004140314b3ddadf268fbc5943d549c948a26a2fe000a7c4"} Oct 30 21:14:30.091821 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:30.091784 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-587445bd6c-djzq5"] Oct 30 21:14:30.096154 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:30.096123 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-587445bd6c-djzq5" Oct 30 21:14:30.211592 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:30.211559 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90ae16e6-1966-43fa-b63d-c83cd50a93c0-tmp\") pod \"console-587445bd6c-djzq5\" (UID: \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\") " pod="openshift-console/console-587445bd6c-djzq5" Oct 30 21:14:30.211778 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:30.211610 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/90ae16e6-1966-43fa-b63d-c83cd50a93c0-oauth-serving-cert\") pod \"console-587445bd6c-djzq5\" (UID: \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\") " pod="openshift-console/console-587445bd6c-djzq5" Oct 30 21:14:30.211778 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:30.211643 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/90ae16e6-1966-43fa-b63d-c83cd50a93c0-console-serving-cert\") pod \"console-587445bd6c-djzq5\" (UID: \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\") " pod="openshift-console/console-587445bd6c-djzq5" Oct 30 21:14:30.211778 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:30.211665 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cskwr\" (UniqueName: \"kubernetes.io/projected/90ae16e6-1966-43fa-b63d-c83cd50a93c0-kube-api-access-cskwr\") pod \"console-587445bd6c-djzq5\" (UID: \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\") " pod="openshift-console/console-587445bd6c-djzq5" Oct 30 21:14:30.211778 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:30.211689 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/90ae16e6-1966-43fa-b63d-c83cd50a93c0-console-config\") pod \"console-587445bd6c-djzq5\" (UID: \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\") " pod="openshift-console/console-587445bd6c-djzq5" Oct 30 21:14:30.211778 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:30.211709 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/90ae16e6-1966-43fa-b63d-c83cd50a93c0-service-ca\") pod \"console-587445bd6c-djzq5\" (UID: \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\") " pod="openshift-console/console-587445bd6c-djzq5" Oct 30 21:14:30.211778 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:30.211738 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90ae16e6-1966-43fa-b63d-c83cd50a93c0-trusted-ca-bundle\") pod \"console-587445bd6c-djzq5\" (UID: \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\") " pod="openshift-console/console-587445bd6c-djzq5" Oct 30 21:14:30.211778 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:30.211761 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/90ae16e6-1966-43fa-b63d-c83cd50a93c0-console-oauth-config\") pod \"console-587445bd6c-djzq5\" (UID: \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\") " pod="openshift-console/console-587445bd6c-djzq5" Oct 30 21:14:30.286282 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:30.286233 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 21:14:30.287179 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:30.287147 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-587445bd6c-djzq5"] Oct 30 21:14:30.312575 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:30.312550 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/90ae16e6-1966-43fa-b63d-c83cd50a93c0-oauth-serving-cert\") pod \"console-587445bd6c-djzq5\" (UID: \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\") " pod="openshift-console/console-587445bd6c-djzq5" Oct 30 21:14:30.312707 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:30.312609 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/90ae16e6-1966-43fa-b63d-c83cd50a93c0-console-serving-cert\") pod \"console-587445bd6c-djzq5\" (UID: \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\") " pod="openshift-console/console-587445bd6c-djzq5" Oct 30 21:14:30.312707 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:30.312631 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cskwr\" (UniqueName: \"kubernetes.io/projected/90ae16e6-1966-43fa-b63d-c83cd50a93c0-kube-api-access-cskwr\") pod \"console-587445bd6c-djzq5\" (UID: \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\") " pod="openshift-console/console-587445bd6c-djzq5" Oct 30 21:14:30.312812 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:30.312795 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/90ae16e6-1966-43fa-b63d-c83cd50a93c0-console-config\") pod \"console-587445bd6c-djzq5\" (UID: \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\") " pod="openshift-console/console-587445bd6c-djzq5" Oct 30 21:14:30.312867 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:30.312826 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/90ae16e6-1966-43fa-b63d-c83cd50a93c0-service-ca\") pod \"console-587445bd6c-djzq5\" (UID: \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\") " pod="openshift-console/console-587445bd6c-djzq5" Oct 30 21:14:30.312867 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:30.312860 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90ae16e6-1966-43fa-b63d-c83cd50a93c0-trusted-ca-bundle\") pod \"console-587445bd6c-djzq5\" (UID: \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\") " pod="openshift-console/console-587445bd6c-djzq5" Oct 30 21:14:30.312971 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:30.312900 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/90ae16e6-1966-43fa-b63d-c83cd50a93c0-console-oauth-config\") pod \"console-587445bd6c-djzq5\" (UID: \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\") " pod="openshift-console/console-587445bd6c-djzq5" Oct 30 21:14:30.312971 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:30.312920 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90ae16e6-1966-43fa-b63d-c83cd50a93c0-tmp\") pod \"console-587445bd6c-djzq5\" (UID: \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\") " pod="openshift-console/console-587445bd6c-djzq5" Oct 30 21:14:30.313303 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:30.313281 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90ae16e6-1966-43fa-b63d-c83cd50a93c0-tmp\") pod \"console-587445bd6c-djzq5\" (UID: \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\") " pod="openshift-console/console-587445bd6c-djzq5" Oct 30 21:14:30.313541 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:30.313519 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/90ae16e6-1966-43fa-b63d-c83cd50a93c0-service-ca\") pod \"console-587445bd6c-djzq5\" (UID: \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\") " pod="openshift-console/console-587445bd6c-djzq5" Oct 30 21:14:30.313622 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:30.313524 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/90ae16e6-1966-43fa-b63d-c83cd50a93c0-console-config\") pod \"console-587445bd6c-djzq5\" (UID: \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\") " pod="openshift-console/console-587445bd6c-djzq5" Oct 30 21:14:30.313830 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:30.313808 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90ae16e6-1966-43fa-b63d-c83cd50a93c0-trusted-ca-bundle\") pod \"console-587445bd6c-djzq5\" (UID: \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\") " pod="openshift-console/console-587445bd6c-djzq5" Oct 30 21:14:30.315713 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:30.315690 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/90ae16e6-1966-43fa-b63d-c83cd50a93c0-console-serving-cert\") pod \"console-587445bd6c-djzq5\" (UID: \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\") " pod="openshift-console/console-587445bd6c-djzq5" Oct 30 21:14:30.315788 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:30.315723 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/90ae16e6-1966-43fa-b63d-c83cd50a93c0-console-oauth-config\") pod \"console-587445bd6c-djzq5\" (UID: \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\") " pod="openshift-console/console-587445bd6c-djzq5" Oct 30 21:14:30.315886 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:30.315858 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/90ae16e6-1966-43fa-b63d-c83cd50a93c0-oauth-serving-cert\") pod \"console-587445bd6c-djzq5\" (UID: \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\") " pod="openshift-console/console-587445bd6c-djzq5" Oct 30 21:14:30.371089 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:30.371023 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cskwr\" (UniqueName: \"kubernetes.io/projected/90ae16e6-1966-43fa-b63d-c83cd50a93c0-kube-api-access-cskwr\") pod \"console-587445bd6c-djzq5\" (UID: \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\") " pod="openshift-console/console-587445bd6c-djzq5" Oct 30 21:14:30.409514 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:30.409485 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-587445bd6c-djzq5" Oct 30 21:14:30.781835 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:30.781807 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c89668d74-8fx8h" event={"ID":"85d7dd2c-56fb-404b-9e24-42413e0b18c9","Type":"ContainerStarted","Data":"8d314b23355f747347224721bc10cc0a1bcc2e5c2b05f00e36126a7aeb29f08b"} Oct 30 21:14:30.786697 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:30.786673 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-587445bd6c-djzq5"] Oct 30 21:14:30.793944 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:14:30.793922 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90ae16e6_1966_43fa_b63d_c83cd50a93c0.slice/crio-2767ec639f575b100fa738ce3c2cb8febe9ce2c93a83a7dfdbc3ca4240b99eac WatchSource:0}: Error finding container 2767ec639f575b100fa738ce3c2cb8febe9ce2c93a83a7dfdbc3ca4240b99eac: Status 404 returned error can't find the container with id 2767ec639f575b100fa738ce3c2cb8febe9ce2c93a83a7dfdbc3ca4240b99eac Oct 30 21:14:30.896130 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:30.896079 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5c89668d74-8fx8h" podStartSLOduration=1.6714805209999999 podStartE2EDuration="4.896064183s" totalImagesPullingTime="3.224583662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:30.895406549 +0000 UTC m=+46.450106327" watchObservedRunningTime="2026-10-30 21:14:30.896064183 +0000 UTC m=+46.450764030" Oct 30 21:14:31.787093 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:31.787054 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-587445bd6c-djzq5" event={"ID":"90ae16e6-1966-43fa-b63d-c83cd50a93c0","Type":"ContainerStarted","Data":"8ec3df52053c2eff66e5a8c31bb4a5715d57ec27c2becb854488910c6ca2d80e"} Oct 30 21:14:31.787499 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:31.787105 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-587445bd6c-djzq5" event={"ID":"90ae16e6-1966-43fa-b63d-c83cd50a93c0","Type":"ContainerStarted","Data":"2767ec639f575b100fa738ce3c2cb8febe9ce2c93a83a7dfdbc3ca4240b99eac"} Oct 30 21:14:31.900002 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:31.899949 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-587445bd6c-djzq5" podStartSLOduration=1.899935642 podStartE2EDuration="1.899935642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:31.898983951 +0000 UTC m=+47.453683728" watchObservedRunningTime="2026-10-30 21:14:31.899935642 +0000 UTC m=+47.454635482" Oct 30 21:14:36.899938 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:36.899894 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c89668d74-8fx8h" Oct 30 21:14:36.899938 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:36.899951 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c89668d74-8fx8h" Oct 30 21:14:36.904763 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:36.904743 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c89668d74-8fx8h" Oct 30 21:14:37.809220 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:37.809188 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c89668d74-8fx8h" Oct 30 21:14:37.929668 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:37.929640 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-756b9cccf4-cz4sx" Oct 30 21:14:37.930036 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:37.929702 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-756b9cccf4-cz4sx" Oct 30 21:14:40.409629 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:40.409590 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-587445bd6c-djzq5" Oct 30 21:14:40.410075 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:40.409656 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-587445bd6c-djzq5" Oct 30 21:14:40.414527 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:40.414504 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-587445bd6c-djzq5" Oct 30 21:14:40.818638 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:40.818610 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-587445bd6c-djzq5" Oct 30 21:14:40.977945 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:40.977912 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c89668d74-8fx8h"] Oct 30 21:14:49.682892 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:14:49.682844 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:14:50.708131 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:50.708098 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-89mnx-pncws"] Oct 30 21:14:50.708590 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:50.708302 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-89mnx-pncws" podUID="f285a424-0cc2-41b8-99b9-92db326ea01d" containerName="insights-gathering" containerID="cri-o://dd90042e5200ab054b9455faf195f596928050d38efdde07a6402d271da4f0bc" gracePeriod=30 Oct 30 21:14:50.852179 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:50.852145 4560 generic.go:356] "Generic (PLEG): container finished" podID="f285a424-0cc2-41b8-99b9-92db326ea01d" containerID="dd90042e5200ab054b9455faf195f596928050d38efdde07a6402d271da4f0bc" exitCode=2 Oct 30 21:14:50.852324 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:50.852216 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-89mnx-pncws" event={"ID":"f285a424-0cc2-41b8-99b9-92db326ea01d","Type":"ContainerDied","Data":"dd90042e5200ab054b9455faf195f596928050d38efdde07a6402d271da4f0bc"} Oct 30 21:14:50.970999 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:50.970952 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-89mnx-pncws" Oct 30 21:14:51.059388 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:51.059359 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f285a424-0cc2-41b8-99b9-92db326ea01d-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f285a424-0cc2-41b8-99b9-92db326ea01d-trusted-ca-bundle\") pod \"f285a424-0cc2-41b8-99b9-92db326ea01d\" (UID: \"f285a424-0cc2-41b8-99b9-92db326ea01d\") " Oct 30 21:14:51.059532 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:51.059453 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f285a424-0cc2-41b8-99b9-92db326ea01d-kube-api-access-gl5cb\" (UniqueName: \"kubernetes.io/projected/f285a424-0cc2-41b8-99b9-92db326ea01d-kube-api-access-gl5cb\") pod \"f285a424-0cc2-41b8-99b9-92db326ea01d\" (UID: \"f285a424-0cc2-41b8-99b9-92db326ea01d\") " Oct 30 21:14:51.059532 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:51.059496 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f285a424-0cc2-41b8-99b9-92db326ea01d-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f285a424-0cc2-41b8-99b9-92db326ea01d-service-ca-bundle\") pod \"f285a424-0cc2-41b8-99b9-92db326ea01d\" (UID: \"f285a424-0cc2-41b8-99b9-92db326ea01d\") " Oct 30 21:14:51.059532 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:51.059522 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f285a424-0cc2-41b8-99b9-92db326ea01d-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f285a424-0cc2-41b8-99b9-92db326ea01d-archives-path\") pod \"f285a424-0cc2-41b8-99b9-92db326ea01d\" (UID: \"f285a424-0cc2-41b8-99b9-92db326ea01d\") " Oct 30 21:14:51.059817 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:51.059795 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f285a424-0cc2-41b8-99b9-92db326ea01d-trusted-ca-bundle" pod "f285a424-0cc2-41b8-99b9-92db326ea01d" (UID: "f285a424-0cc2-41b8-99b9-92db326ea01d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:14:51.059817 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:51.059805 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f285a424-0cc2-41b8-99b9-92db326ea01d-service-ca-bundle" pod "f285a424-0cc2-41b8-99b9-92db326ea01d" (UID: "f285a424-0cc2-41b8-99b9-92db326ea01d"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:14:51.059890 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:51.059876 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f285a424-0cc2-41b8-99b9-92db326ea01d-archives-path" pod "f285a424-0cc2-41b8-99b9-92db326ea01d" (UID: "f285a424-0cc2-41b8-99b9-92db326ea01d"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:51.061661 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:51.061638 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f285a424-0cc2-41b8-99b9-92db326ea01d-kube-api-access-gl5cb" pod "f285a424-0cc2-41b8-99b9-92db326ea01d" (UID: "f285a424-0cc2-41b8-99b9-92db326ea01d"). InnerVolumeSpecName "kube-api-access-gl5cb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:14:51.160389 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:51.160330 4560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f285a424-0cc2-41b8-99b9-92db326ea01d-trusted-ca-bundle\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:51.160389 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:51.160382 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl5cb\" (UniqueName: \"kubernetes.io/projected/f285a424-0cc2-41b8-99b9-92db326ea01d-kube-api-access-gl5cb\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:51.160389 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:51.160394 4560 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f285a424-0cc2-41b8-99b9-92db326ea01d-service-ca-bundle\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:51.160617 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:51.160402 4560 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f285a424-0cc2-41b8-99b9-92db326ea01d-archives-path\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:51.856775 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:51.856743 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-89mnx-pncws" event={"ID":"f285a424-0cc2-41b8-99b9-92db326ea01d","Type":"ContainerDied","Data":"8312a77f41c875aae9eea8ead18164c2574c4b690333c9efaae39ac09e4f37e0"} Oct 30 21:14:51.856775 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:51.856774 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-89mnx-pncws" Oct 30 21:14:51.857245 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:51.856812 4560 scope.go:122] "RemoveContainer" containerID="dd90042e5200ab054b9455faf195f596928050d38efdde07a6402d271da4f0bc" Oct 30 21:14:52.010809 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:52.010782 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-89mnx-pncws"] Oct 30 21:14:52.057305 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:52.057282 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-89mnx-pncws"] Oct 30 21:14:52.572668 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:52.572637 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f285a424-0cc2-41b8-99b9-92db326ea01d" path="/var/lib/kubelet/pods/f285a424-0cc2-41b8-99b9-92db326ea01d/volumes" Oct 30 21:14:57.934163 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:57.934139 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-756b9cccf4-cz4sx" Oct 30 21:14:57.937954 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:14:57.937937 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-756b9cccf4-cz4sx" Oct 30 21:15:05.997724 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:05.997677 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5c89668d74-8fx8h" podUID="85d7dd2c-56fb-404b-9e24-42413e0b18c9" containerName="console" containerID="cri-o://8d314b23355f747347224721bc10cc0a1bcc2e5c2b05f00e36126a7aeb29f08b" gracePeriod=15 Oct 30 21:15:06.257543 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:06.257491 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c89668d74-8fx8h" Oct 30 21:15:06.366137 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:06.366107 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85d7dd2c-56fb-404b-9e24-42413e0b18c9-service-ca\" (UniqueName: \"kubernetes.io/configmap/85d7dd2c-56fb-404b-9e24-42413e0b18c9-service-ca\") pod \"85d7dd2c-56fb-404b-9e24-42413e0b18c9\" (UID: \"85d7dd2c-56fb-404b-9e24-42413e0b18c9\") " Oct 30 21:15:06.366255 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:06.366155 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85d7dd2c-56fb-404b-9e24-42413e0b18c9-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85d7dd2c-56fb-404b-9e24-42413e0b18c9-oauth-serving-cert\") pod \"85d7dd2c-56fb-404b-9e24-42413e0b18c9\" (UID: \"85d7dd2c-56fb-404b-9e24-42413e0b18c9\") " Oct 30 21:15:06.366255 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:06.366177 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85d7dd2c-56fb-404b-9e24-42413e0b18c9-console-config\" (UniqueName: \"kubernetes.io/configmap/85d7dd2c-56fb-404b-9e24-42413e0b18c9-console-config\") pod \"85d7dd2c-56fb-404b-9e24-42413e0b18c9\" (UID: \"85d7dd2c-56fb-404b-9e24-42413e0b18c9\") " Oct 30 21:15:06.366255 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:06.366220 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85d7dd2c-56fb-404b-9e24-42413e0b18c9-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85d7dd2c-56fb-404b-9e24-42413e0b18c9-console-oauth-config\") pod \"85d7dd2c-56fb-404b-9e24-42413e0b18c9\" (UID: \"85d7dd2c-56fb-404b-9e24-42413e0b18c9\") " Oct 30 21:15:06.366255 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:06.366246 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85d7dd2c-56fb-404b-9e24-42413e0b18c9-tmp\" (UniqueName: \"kubernetes.io/empty-dir/85d7dd2c-56fb-404b-9e24-42413e0b18c9-tmp\") pod \"85d7dd2c-56fb-404b-9e24-42413e0b18c9\" (UID: \"85d7dd2c-56fb-404b-9e24-42413e0b18c9\") " Oct 30 21:15:06.366478 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:06.366266 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85d7dd2c-56fb-404b-9e24-42413e0b18c9-kube-api-access-hgndk\" (UniqueName: \"kubernetes.io/projected/85d7dd2c-56fb-404b-9e24-42413e0b18c9-kube-api-access-hgndk\") pod \"85d7dd2c-56fb-404b-9e24-42413e0b18c9\" (UID: \"85d7dd2c-56fb-404b-9e24-42413e0b18c9\") " Oct 30 21:15:06.366478 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:06.366290 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85d7dd2c-56fb-404b-9e24-42413e0b18c9-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85d7dd2c-56fb-404b-9e24-42413e0b18c9-console-serving-cert\") pod \"85d7dd2c-56fb-404b-9e24-42413e0b18c9\" (UID: \"85d7dd2c-56fb-404b-9e24-42413e0b18c9\") " Oct 30 21:15:06.366682 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:06.366640 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85d7dd2c-56fb-404b-9e24-42413e0b18c9-oauth-serving-cert" pod "85d7dd2c-56fb-404b-9e24-42413e0b18c9" (UID: "85d7dd2c-56fb-404b-9e24-42413e0b18c9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:15:06.366682 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:06.366623 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d7dd2c-56fb-404b-9e24-42413e0b18c9-tmp" pod "85d7dd2c-56fb-404b-9e24-42413e0b18c9" (UID: "85d7dd2c-56fb-404b-9e24-42413e0b18c9"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:15:06.366839 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:06.366686 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85d7dd2c-56fb-404b-9e24-42413e0b18c9-service-ca" pod "85d7dd2c-56fb-404b-9e24-42413e0b18c9" (UID: "85d7dd2c-56fb-404b-9e24-42413e0b18c9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:15:06.366839 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:06.366751 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85d7dd2c-56fb-404b-9e24-42413e0b18c9-console-config" pod "85d7dd2c-56fb-404b-9e24-42413e0b18c9" (UID: "85d7dd2c-56fb-404b-9e24-42413e0b18c9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:15:06.368586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:06.368557 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85d7dd2c-56fb-404b-9e24-42413e0b18c9-console-serving-cert" pod "85d7dd2c-56fb-404b-9e24-42413e0b18c9" (UID: "85d7dd2c-56fb-404b-9e24-42413e0b18c9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:15:06.368797 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:06.368764 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85d7dd2c-56fb-404b-9e24-42413e0b18c9-kube-api-access-hgndk" pod "85d7dd2c-56fb-404b-9e24-42413e0b18c9" (UID: "85d7dd2c-56fb-404b-9e24-42413e0b18c9"). InnerVolumeSpecName "kube-api-access-hgndk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:15:06.368958 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:06.368792 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85d7dd2c-56fb-404b-9e24-42413e0b18c9-console-oauth-config" pod "85d7dd2c-56fb-404b-9e24-42413e0b18c9" (UID: "85d7dd2c-56fb-404b-9e24-42413e0b18c9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:15:06.467457 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:06.467412 4560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85d7dd2c-56fb-404b-9e24-42413e0b18c9-service-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:15:06.467457 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:06.467452 4560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85d7dd2c-56fb-404b-9e24-42413e0b18c9-oauth-serving-cert\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:15:06.467457 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:06.467462 4560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85d7dd2c-56fb-404b-9e24-42413e0b18c9-console-config\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:15:06.467457 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:06.467471 4560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85d7dd2c-56fb-404b-9e24-42413e0b18c9-console-oauth-config\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:15:06.467719 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:06.467480 4560 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85d7dd2c-56fb-404b-9e24-42413e0b18c9-tmp\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:15:06.467719 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:06.467491 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgndk\" (UniqueName: \"kubernetes.io/projected/85d7dd2c-56fb-404b-9e24-42413e0b18c9-kube-api-access-hgndk\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:15:06.467719 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:06.467500 4560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85d7dd2c-56fb-404b-9e24-42413e0b18c9-console-serving-cert\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:15:06.904812 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:06.904719 4560 generic.go:356] "Generic (PLEG): container finished" podID="85d7dd2c-56fb-404b-9e24-42413e0b18c9" containerID="8d314b23355f747347224721bc10cc0a1bcc2e5c2b05f00e36126a7aeb29f08b" exitCode=0 Oct 30 21:15:06.904812 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:06.904775 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c89668d74-8fx8h" Oct 30 21:15:06.904999 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:06.904805 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c89668d74-8fx8h" event={"ID":"85d7dd2c-56fb-404b-9e24-42413e0b18c9","Type":"ContainerDied","Data":"8d314b23355f747347224721bc10cc0a1bcc2e5c2b05f00e36126a7aeb29f08b"} Oct 30 21:15:06.904999 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:06.904854 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c89668d74-8fx8h" event={"ID":"85d7dd2c-56fb-404b-9e24-42413e0b18c9","Type":"ContainerDied","Data":"3c8ea08172cbbeb7004140314b3ddadf268fbc5943d549c948a26a2fe000a7c4"} Oct 30 21:15:06.904999 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:06.904874 4560 scope.go:122] "RemoveContainer" containerID="8d314b23355f747347224721bc10cc0a1bcc2e5c2b05f00e36126a7aeb29f08b" Oct 30 21:15:06.912951 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:06.912930 4560 scope.go:122] "RemoveContainer" containerID="8d314b23355f747347224721bc10cc0a1bcc2e5c2b05f00e36126a7aeb29f08b" Oct 30 21:15:06.913236 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:15:06.913217 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d314b23355f747347224721bc10cc0a1bcc2e5c2b05f00e36126a7aeb29f08b\": container with ID starting with 8d314b23355f747347224721bc10cc0a1bcc2e5c2b05f00e36126a7aeb29f08b not found: ID does not exist" containerID="8d314b23355f747347224721bc10cc0a1bcc2e5c2b05f00e36126a7aeb29f08b" Oct 30 21:15:06.913284 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:06.913245 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d314b23355f747347224721bc10cc0a1bcc2e5c2b05f00e36126a7aeb29f08b"} err="failed to get container status \"8d314b23355f747347224721bc10cc0a1bcc2e5c2b05f00e36126a7aeb29f08b\": rpc error: code = NotFound desc = could not find container \"8d314b23355f747347224721bc10cc0a1bcc2e5c2b05f00e36126a7aeb29f08b\": container with ID starting with 8d314b23355f747347224721bc10cc0a1bcc2e5c2b05f00e36126a7aeb29f08b not found: ID does not exist" Oct 30 21:15:07.059091 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:07.059059 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c89668d74-8fx8h"] Oct 30 21:15:07.105812 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:07.105775 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c89668d74-8fx8h"] Oct 30 21:15:08.572762 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:08.572721 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85d7dd2c-56fb-404b-9e24-42413e0b18c9" path="/var/lib/kubelet/pods/85d7dd2c-56fb-404b-9e24-42413e0b18c9/volumes" Oct 30 21:15:19.682334 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:15:19.682298 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:15:36.837225 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:36.837187 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d6bfcf686-w5txk"] Oct 30 21:15:36.837721 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:36.837484 4560 cpu_manager.go:414] "containerMap: removing container" podUID="f285a424-0cc2-41b8-99b9-92db326ea01d" containerName="insights-gathering" Oct 30 21:15:36.837721 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:36.837498 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f285a424-0cc2-41b8-99b9-92db326ea01d" containerName="insights-gathering" Oct 30 21:15:36.837721 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:36.837508 4560 cpu_manager.go:414] "containerMap: removing container" podUID="85d7dd2c-56fb-404b-9e24-42413e0b18c9" containerName="console" Oct 30 21:15:36.837721 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:36.837513 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85d7dd2c-56fb-404b-9e24-42413e0b18c9" containerName="console" Oct 30 21:15:36.837721 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:36.837570 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="f285a424-0cc2-41b8-99b9-92db326ea01d" containerName="insights-gathering" Oct 30 21:15:36.837721 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:36.837580 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="85d7dd2c-56fb-404b-9e24-42413e0b18c9" containerName="console" Oct 30 21:15:36.839564 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:36.839548 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d6bfcf686-w5txk" Oct 30 21:15:36.868985 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:36.868958 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/392b99e3-42e2-4bc9-9421-917b4cb5b83b-console-oauth-config\") pod \"console-5d6bfcf686-w5txk\" (UID: \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\") " pod="openshift-console/console-5d6bfcf686-w5txk" Oct 30 21:15:36.868985 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:36.868986 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/392b99e3-42e2-4bc9-9421-917b4cb5b83b-tmp\") pod \"console-5d6bfcf686-w5txk\" (UID: \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\") " pod="openshift-console/console-5d6bfcf686-w5txk" Oct 30 21:15:36.869131 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:36.869012 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/392b99e3-42e2-4bc9-9421-917b4cb5b83b-oauth-serving-cert\") pod \"console-5d6bfcf686-w5txk\" (UID: \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\") " pod="openshift-console/console-5d6bfcf686-w5txk" Oct 30 21:15:36.869131 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:36.869064 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/392b99e3-42e2-4bc9-9421-917b4cb5b83b-console-config\") pod \"console-5d6bfcf686-w5txk\" (UID: \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\") " pod="openshift-console/console-5d6bfcf686-w5txk" Oct 30 21:15:36.869131 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:36.869091 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8jkq\" (UniqueName: \"kubernetes.io/projected/392b99e3-42e2-4bc9-9421-917b4cb5b83b-kube-api-access-f8jkq\") pod \"console-5d6bfcf686-w5txk\" (UID: \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\") " pod="openshift-console/console-5d6bfcf686-w5txk" Oct 30 21:15:36.869240 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:36.869175 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/392b99e3-42e2-4bc9-9421-917b4cb5b83b-console-serving-cert\") pod \"console-5d6bfcf686-w5txk\" (UID: \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\") " pod="openshift-console/console-5d6bfcf686-w5txk" Oct 30 21:15:36.869240 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:36.869200 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/392b99e3-42e2-4bc9-9421-917b4cb5b83b-service-ca\") pod \"console-5d6bfcf686-w5txk\" (UID: \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\") " pod="openshift-console/console-5d6bfcf686-w5txk" Oct 30 21:15:36.869240 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:36.869217 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/392b99e3-42e2-4bc9-9421-917b4cb5b83b-trusted-ca-bundle\") pod \"console-5d6bfcf686-w5txk\" (UID: \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\") " pod="openshift-console/console-5d6bfcf686-w5txk" Oct 30 21:15:36.935859 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:36.935828 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d6bfcf686-w5txk"] Oct 30 21:15:36.970497 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:36.970469 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/392b99e3-42e2-4bc9-9421-917b4cb5b83b-oauth-serving-cert\") pod \"console-5d6bfcf686-w5txk\" (UID: \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\") " pod="openshift-console/console-5d6bfcf686-w5txk" Oct 30 21:15:36.970611 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:36.970508 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/392b99e3-42e2-4bc9-9421-917b4cb5b83b-console-config\") pod \"console-5d6bfcf686-w5txk\" (UID: \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\") " pod="openshift-console/console-5d6bfcf686-w5txk" Oct 30 21:15:36.970611 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:36.970531 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8jkq\" (UniqueName: \"kubernetes.io/projected/392b99e3-42e2-4bc9-9421-917b4cb5b83b-kube-api-access-f8jkq\") pod \"console-5d6bfcf686-w5txk\" (UID: \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\") " pod="openshift-console/console-5d6bfcf686-w5txk" Oct 30 21:15:36.970611 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:36.970575 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/392b99e3-42e2-4bc9-9421-917b4cb5b83b-console-serving-cert\") pod \"console-5d6bfcf686-w5txk\" (UID: \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\") " pod="openshift-console/console-5d6bfcf686-w5txk" Oct 30 21:15:36.970775 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:36.970614 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/392b99e3-42e2-4bc9-9421-917b4cb5b83b-service-ca\") pod \"console-5d6bfcf686-w5txk\" (UID: \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\") " pod="openshift-console/console-5d6bfcf686-w5txk" Oct 30 21:15:36.970775 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:36.970639 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/392b99e3-42e2-4bc9-9421-917b4cb5b83b-trusted-ca-bundle\") pod \"console-5d6bfcf686-w5txk\" (UID: \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\") " pod="openshift-console/console-5d6bfcf686-w5txk" Oct 30 21:15:36.970775 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:36.970676 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/392b99e3-42e2-4bc9-9421-917b4cb5b83b-console-oauth-config\") pod \"console-5d6bfcf686-w5txk\" (UID: \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\") " pod="openshift-console/console-5d6bfcf686-w5txk" Oct 30 21:15:36.970775 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:36.970703 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/392b99e3-42e2-4bc9-9421-917b4cb5b83b-tmp\") pod \"console-5d6bfcf686-w5txk\" (UID: \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\") " pod="openshift-console/console-5d6bfcf686-w5txk" Oct 30 21:15:36.971092 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:36.971067 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/392b99e3-42e2-4bc9-9421-917b4cb5b83b-tmp\") pod \"console-5d6bfcf686-w5txk\" (UID: \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\") " pod="openshift-console/console-5d6bfcf686-w5txk" Oct 30 21:15:36.971324 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:36.971304 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/392b99e3-42e2-4bc9-9421-917b4cb5b83b-oauth-serving-cert\") pod \"console-5d6bfcf686-w5txk\" (UID: \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\") " pod="openshift-console/console-5d6bfcf686-w5txk" Oct 30 21:15:36.971422 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:36.971329 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/392b99e3-42e2-4bc9-9421-917b4cb5b83b-service-ca\") pod \"console-5d6bfcf686-w5txk\" (UID: \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\") " pod="openshift-console/console-5d6bfcf686-w5txk" Oct 30 21:15:36.971560 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:36.971542 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/392b99e3-42e2-4bc9-9421-917b4cb5b83b-console-config\") pod \"console-5d6bfcf686-w5txk\" (UID: \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\") " pod="openshift-console/console-5d6bfcf686-w5txk" Oct 30 21:15:36.971651 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:36.971632 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/392b99e3-42e2-4bc9-9421-917b4cb5b83b-trusted-ca-bundle\") pod \"console-5d6bfcf686-w5txk\" (UID: \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\") " pod="openshift-console/console-5d6bfcf686-w5txk" Oct 30 21:15:36.973045 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:36.973025 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/392b99e3-42e2-4bc9-9421-917b4cb5b83b-console-oauth-config\") pod \"console-5d6bfcf686-w5txk\" (UID: \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\") " pod="openshift-console/console-5d6bfcf686-w5txk" Oct 30 21:15:36.973290 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:36.973269 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/392b99e3-42e2-4bc9-9421-917b4cb5b83b-console-serving-cert\") pod \"console-5d6bfcf686-w5txk\" (UID: \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\") " pod="openshift-console/console-5d6bfcf686-w5txk" Oct 30 21:15:37.032137 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:37.032108 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8jkq\" (UniqueName: \"kubernetes.io/projected/392b99e3-42e2-4bc9-9421-917b4cb5b83b-kube-api-access-f8jkq\") pod \"console-5d6bfcf686-w5txk\" (UID: \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\") " pod="openshift-console/console-5d6bfcf686-w5txk" Oct 30 21:15:37.149776 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:37.149699 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d6bfcf686-w5txk" Oct 30 21:15:37.517609 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:37.517574 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d6bfcf686-w5txk"] Oct 30 21:15:37.524205 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:15:37.524170 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod392b99e3_42e2_4bc9_9421_917b4cb5b83b.slice/crio-126fbc35959cda6b6157287f30c6373c2c76e4758c33cc52fd84806430519d46 WatchSource:0}: Error finding container 126fbc35959cda6b6157287f30c6373c2c76e4758c33cc52fd84806430519d46: Status 404 returned error can't find the container with id 126fbc35959cda6b6157287f30c6373c2c76e4758c33cc52fd84806430519d46 Oct 30 21:15:38.002799 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:38.002763 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d6bfcf686-w5txk" event={"ID":"392b99e3-42e2-4bc9-9421-917b4cb5b83b","Type":"ContainerStarted","Data":"94c5c6601e9a7401821d744b503baf6c0d4971cf5f8df1b2d6a2ab5c738ecea3"} Oct 30 21:15:38.003162 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:38.002818 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d6bfcf686-w5txk" event={"ID":"392b99e3-42e2-4bc9-9421-917b4cb5b83b","Type":"ContainerStarted","Data":"126fbc35959cda6b6157287f30c6373c2c76e4758c33cc52fd84806430519d46"} Oct 30 21:15:38.115206 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:38.115157 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5d6bfcf686-w5txk" podStartSLOduration=2.115142294 podStartE2EDuration="2.115142294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:38.114879267 +0000 UTC m=+113.669579046" watchObservedRunningTime="2026-10-30 21:15:38.115142294 +0000 UTC m=+113.669842071" Oct 30 21:15:47.149967 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:47.149931 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d6bfcf686-w5txk" Oct 30 21:15:47.150362 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:47.149992 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d6bfcf686-w5txk" Oct 30 21:15:47.154553 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:47.154534 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d6bfcf686-w5txk" Oct 30 21:15:48.037972 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:48.037944 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d6bfcf686-w5txk" Oct 30 21:15:48.197655 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:15:48.197621 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-587445bd6c-djzq5"] Oct 30 21:15:49.682600 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:15:49.682564 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:16:13.216590 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:13.216505 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-587445bd6c-djzq5" podUID="90ae16e6-1966-43fa-b63d-c83cd50a93c0" containerName="console" containerID="cri-o://8ec3df52053c2eff66e5a8c31bb4a5715d57ec27c2becb854488910c6ca2d80e" gracePeriod=15 Oct 30 21:16:13.460752 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:13.460725 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-587445bd6c-djzq5" Oct 30 21:16:13.536535 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:13.536508 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90ae16e6-1966-43fa-b63d-c83cd50a93c0-console-config\" (UniqueName: \"kubernetes.io/configmap/90ae16e6-1966-43fa-b63d-c83cd50a93c0-console-config\") pod \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\" (UID: \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\") " Oct 30 21:16:13.536723 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:13.536553 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90ae16e6-1966-43fa-b63d-c83cd50a93c0-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/90ae16e6-1966-43fa-b63d-c83cd50a93c0-console-serving-cert\") pod \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\" (UID: \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\") " Oct 30 21:16:13.536723 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:13.536573 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90ae16e6-1966-43fa-b63d-c83cd50a93c0-service-ca\" (UniqueName: \"kubernetes.io/configmap/90ae16e6-1966-43fa-b63d-c83cd50a93c0-service-ca\") pod \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\" (UID: \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\") " Oct 30 21:16:13.536723 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:13.536618 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90ae16e6-1966-43fa-b63d-c83cd50a93c0-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/90ae16e6-1966-43fa-b63d-c83cd50a93c0-oauth-serving-cert\") pod \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\" (UID: \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\") " Oct 30 21:16:13.536723 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:13.536639 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90ae16e6-1966-43fa-b63d-c83cd50a93c0-tmp\" (UniqueName: \"kubernetes.io/empty-dir/90ae16e6-1966-43fa-b63d-c83cd50a93c0-tmp\") pod \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\" (UID: \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\") " Oct 30 21:16:13.536723 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:13.536677 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90ae16e6-1966-43fa-b63d-c83cd50a93c0-kube-api-access-cskwr\" (UniqueName: \"kubernetes.io/projected/90ae16e6-1966-43fa-b63d-c83cd50a93c0-kube-api-access-cskwr\") pod \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\" (UID: \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\") " Oct 30 21:16:13.536723 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:13.536716 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90ae16e6-1966-43fa-b63d-c83cd50a93c0-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/90ae16e6-1966-43fa-b63d-c83cd50a93c0-console-oauth-config\") pod \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\" (UID: \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\") " Oct 30 21:16:13.537019 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:13.536764 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90ae16e6-1966-43fa-b63d-c83cd50a93c0-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90ae16e6-1966-43fa-b63d-c83cd50a93c0-trusted-ca-bundle\") pod \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\" (UID: \"90ae16e6-1966-43fa-b63d-c83cd50a93c0\") " Oct 30 21:16:13.537019 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:13.536926 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90ae16e6-1966-43fa-b63d-c83cd50a93c0-console-config" pod "90ae16e6-1966-43fa-b63d-c83cd50a93c0" (UID: "90ae16e6-1966-43fa-b63d-c83cd50a93c0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:16:13.537273 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:13.537243 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90ae16e6-1966-43fa-b63d-c83cd50a93c0-trusted-ca-bundle" pod "90ae16e6-1966-43fa-b63d-c83cd50a93c0" (UID: "90ae16e6-1966-43fa-b63d-c83cd50a93c0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:16:13.537431 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:13.537365 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90ae16e6-1966-43fa-b63d-c83cd50a93c0-tmp" pod "90ae16e6-1966-43fa-b63d-c83cd50a93c0" (UID: "90ae16e6-1966-43fa-b63d-c83cd50a93c0"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:16:13.537431 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:13.537383 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90ae16e6-1966-43fa-b63d-c83cd50a93c0-oauth-serving-cert" pod "90ae16e6-1966-43fa-b63d-c83cd50a93c0" (UID: "90ae16e6-1966-43fa-b63d-c83cd50a93c0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:16:13.537431 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:13.537388 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90ae16e6-1966-43fa-b63d-c83cd50a93c0-service-ca" pod "90ae16e6-1966-43fa-b63d-c83cd50a93c0" (UID: "90ae16e6-1966-43fa-b63d-c83cd50a93c0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:16:13.538719 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:13.538694 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90ae16e6-1966-43fa-b63d-c83cd50a93c0-console-serving-cert" pod "90ae16e6-1966-43fa-b63d-c83cd50a93c0" (UID: "90ae16e6-1966-43fa-b63d-c83cd50a93c0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:16:13.538873 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:13.538845 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90ae16e6-1966-43fa-b63d-c83cd50a93c0-console-oauth-config" pod "90ae16e6-1966-43fa-b63d-c83cd50a93c0" (UID: "90ae16e6-1966-43fa-b63d-c83cd50a93c0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:16:13.538915 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:13.538863 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90ae16e6-1966-43fa-b63d-c83cd50a93c0-kube-api-access-cskwr" pod "90ae16e6-1966-43fa-b63d-c83cd50a93c0" (UID: "90ae16e6-1966-43fa-b63d-c83cd50a93c0"). InnerVolumeSpecName "kube-api-access-cskwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:16:13.638125 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:13.638100 4560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/90ae16e6-1966-43fa-b63d-c83cd50a93c0-oauth-serving-cert\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:16:13.638125 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:13.638124 4560 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90ae16e6-1966-43fa-b63d-c83cd50a93c0-tmp\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:16:13.638301 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:13.638134 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cskwr\" (UniqueName: \"kubernetes.io/projected/90ae16e6-1966-43fa-b63d-c83cd50a93c0-kube-api-access-cskwr\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:16:13.638301 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:13.638144 4560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/90ae16e6-1966-43fa-b63d-c83cd50a93c0-console-oauth-config\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:16:13.638301 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:13.638154 4560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90ae16e6-1966-43fa-b63d-c83cd50a93c0-trusted-ca-bundle\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:16:13.638301 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:13.638162 4560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/90ae16e6-1966-43fa-b63d-c83cd50a93c0-console-config\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:16:13.638301 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:13.638171 4560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/90ae16e6-1966-43fa-b63d-c83cd50a93c0-console-serving-cert\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:16:13.638301 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:13.638179 4560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/90ae16e6-1966-43fa-b63d-c83cd50a93c0-service-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:16:14.113076 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:14.113045 4560 generic.go:356] "Generic (PLEG): container finished" podID="90ae16e6-1966-43fa-b63d-c83cd50a93c0" containerID="8ec3df52053c2eff66e5a8c31bb4a5715d57ec27c2becb854488910c6ca2d80e" exitCode=0 Oct 30 21:16:14.113246 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:14.113106 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-587445bd6c-djzq5" Oct 30 21:16:14.113246 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:14.113117 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-587445bd6c-djzq5" event={"ID":"90ae16e6-1966-43fa-b63d-c83cd50a93c0","Type":"ContainerDied","Data":"8ec3df52053c2eff66e5a8c31bb4a5715d57ec27c2becb854488910c6ca2d80e"} Oct 30 21:16:14.113246 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:14.113180 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-587445bd6c-djzq5" event={"ID":"90ae16e6-1966-43fa-b63d-c83cd50a93c0","Type":"ContainerDied","Data":"2767ec639f575b100fa738ce3c2cb8febe9ce2c93a83a7dfdbc3ca4240b99eac"} Oct 30 21:16:14.113246 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:14.113199 4560 scope.go:122] "RemoveContainer" containerID="8ec3df52053c2eff66e5a8c31bb4a5715d57ec27c2becb854488910c6ca2d80e" Oct 30 21:16:14.121909 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:14.121852 4560 scope.go:122] "RemoveContainer" containerID="8ec3df52053c2eff66e5a8c31bb4a5715d57ec27c2becb854488910c6ca2d80e" Oct 30 21:16:14.122208 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:16:14.122187 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ec3df52053c2eff66e5a8c31bb4a5715d57ec27c2becb854488910c6ca2d80e\": container with ID starting with 8ec3df52053c2eff66e5a8c31bb4a5715d57ec27c2becb854488910c6ca2d80e not found: ID does not exist" containerID="8ec3df52053c2eff66e5a8c31bb4a5715d57ec27c2becb854488910c6ca2d80e" Oct 30 21:16:14.122282 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:14.122220 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ec3df52053c2eff66e5a8c31bb4a5715d57ec27c2becb854488910c6ca2d80e"} err="failed to get container status \"8ec3df52053c2eff66e5a8c31bb4a5715d57ec27c2becb854488910c6ca2d80e\": rpc error: code = NotFound desc = could not find container \"8ec3df52053c2eff66e5a8c31bb4a5715d57ec27c2becb854488910c6ca2d80e\": container with ID starting with 8ec3df52053c2eff66e5a8c31bb4a5715d57ec27c2becb854488910c6ca2d80e not found: ID does not exist" Oct 30 21:16:14.268056 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:14.268013 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-587445bd6c-djzq5"] Oct 30 21:16:14.283900 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:14.283865 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-587445bd6c-djzq5"] Oct 30 21:16:14.573006 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:14.572975 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90ae16e6-1966-43fa-b63d-c83cd50a93c0" path="/var/lib/kubelet/pods/90ae16e6-1966-43fa-b63d-c83cd50a93c0/volumes" Oct 30 21:16:19.683805 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:16:19.683767 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:16:49.683447 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:16:49.683406 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:16:54.370729 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:54.370691 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-c89c876c6-6smjv"] Oct 30 21:16:54.371196 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:54.370988 4560 cpu_manager.go:414] "containerMap: removing container" podUID="90ae16e6-1966-43fa-b63d-c83cd50a93c0" containerName="console" Oct 30 21:16:54.371196 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:54.371001 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90ae16e6-1966-43fa-b63d-c83cd50a93c0" containerName="console" Oct 30 21:16:54.371196 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:54.371057 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="90ae16e6-1966-43fa-b63d-c83cd50a93c0" containerName="console" Oct 30 21:16:54.374106 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:54.374089 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c89c876c6-6smjv" Oct 30 21:16:54.415170 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:54.415139 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/283648de-3941-45ac-ada0-2e33e84eac60-console-config\") pod \"console-c89c876c6-6smjv\" (UID: \"283648de-3941-45ac-ada0-2e33e84eac60\") " pod="openshift-console/console-c89c876c6-6smjv" Oct 30 21:16:54.415284 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:54.415174 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/283648de-3941-45ac-ada0-2e33e84eac60-console-oauth-config\") pod \"console-c89c876c6-6smjv\" (UID: \"283648de-3941-45ac-ada0-2e33e84eac60\") " pod="openshift-console/console-c89c876c6-6smjv" Oct 30 21:16:54.415284 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:54.415193 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b65w\" (UniqueName: \"kubernetes.io/projected/283648de-3941-45ac-ada0-2e33e84eac60-kube-api-access-6b65w\") pod \"console-c89c876c6-6smjv\" (UID: \"283648de-3941-45ac-ada0-2e33e84eac60\") " pod="openshift-console/console-c89c876c6-6smjv" Oct 30 21:16:54.415376 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:54.415280 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/283648de-3941-45ac-ada0-2e33e84eac60-oauth-serving-cert\") pod \"console-c89c876c6-6smjv\" (UID: \"283648de-3941-45ac-ada0-2e33e84eac60\") " pod="openshift-console/console-c89c876c6-6smjv" Oct 30 21:16:54.415376 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:54.415322 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/283648de-3941-45ac-ada0-2e33e84eac60-service-ca\") pod \"console-c89c876c6-6smjv\" (UID: \"283648de-3941-45ac-ada0-2e33e84eac60\") " pod="openshift-console/console-c89c876c6-6smjv" Oct 30 21:16:54.415376 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:54.415354 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/283648de-3941-45ac-ada0-2e33e84eac60-tmp\") pod \"console-c89c876c6-6smjv\" (UID: \"283648de-3941-45ac-ada0-2e33e84eac60\") " pod="openshift-console/console-c89c876c6-6smjv" Oct 30 21:16:54.415491 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:54.415377 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/283648de-3941-45ac-ada0-2e33e84eac60-trusted-ca-bundle\") pod \"console-c89c876c6-6smjv\" (UID: \"283648de-3941-45ac-ada0-2e33e84eac60\") " pod="openshift-console/console-c89c876c6-6smjv" Oct 30 21:16:54.415491 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:54.415395 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/283648de-3941-45ac-ada0-2e33e84eac60-console-serving-cert\") pod \"console-c89c876c6-6smjv\" (UID: \"283648de-3941-45ac-ada0-2e33e84eac60\") " pod="openshift-console/console-c89c876c6-6smjv" Oct 30 21:16:54.470774 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:54.470742 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c89c876c6-6smjv"] Oct 30 21:16:54.516191 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:54.516155 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/283648de-3941-45ac-ada0-2e33e84eac60-console-config\") pod \"console-c89c876c6-6smjv\" (UID: \"283648de-3941-45ac-ada0-2e33e84eac60\") " pod="openshift-console/console-c89c876c6-6smjv" Oct 30 21:16:54.516191 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:54.516193 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/283648de-3941-45ac-ada0-2e33e84eac60-console-oauth-config\") pod \"console-c89c876c6-6smjv\" (UID: \"283648de-3941-45ac-ada0-2e33e84eac60\") " pod="openshift-console/console-c89c876c6-6smjv" Oct 30 21:16:54.516477 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:54.516213 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b65w\" (UniqueName: \"kubernetes.io/projected/283648de-3941-45ac-ada0-2e33e84eac60-kube-api-access-6b65w\") pod \"console-c89c876c6-6smjv\" (UID: \"283648de-3941-45ac-ada0-2e33e84eac60\") " pod="openshift-console/console-c89c876c6-6smjv" Oct 30 21:16:54.516477 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:54.516235 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/283648de-3941-45ac-ada0-2e33e84eac60-oauth-serving-cert\") pod \"console-c89c876c6-6smjv\" (UID: \"283648de-3941-45ac-ada0-2e33e84eac60\") " pod="openshift-console/console-c89c876c6-6smjv" Oct 30 21:16:54.516477 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:54.516251 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/283648de-3941-45ac-ada0-2e33e84eac60-service-ca\") pod \"console-c89c876c6-6smjv\" (UID: \"283648de-3941-45ac-ada0-2e33e84eac60\") " pod="openshift-console/console-c89c876c6-6smjv" Oct 30 21:16:54.516477 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:54.516267 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/283648de-3941-45ac-ada0-2e33e84eac60-tmp\") pod \"console-c89c876c6-6smjv\" (UID: \"283648de-3941-45ac-ada0-2e33e84eac60\") " pod="openshift-console/console-c89c876c6-6smjv" Oct 30 21:16:54.516477 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:54.516303 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/283648de-3941-45ac-ada0-2e33e84eac60-trusted-ca-bundle\") pod \"console-c89c876c6-6smjv\" (UID: \"283648de-3941-45ac-ada0-2e33e84eac60\") " pod="openshift-console/console-c89c876c6-6smjv" Oct 30 21:16:54.516477 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:54.516331 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/283648de-3941-45ac-ada0-2e33e84eac60-console-serving-cert\") pod \"console-c89c876c6-6smjv\" (UID: \"283648de-3941-45ac-ada0-2e33e84eac60\") " pod="openshift-console/console-c89c876c6-6smjv" Oct 30 21:16:54.516875 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:54.516849 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/283648de-3941-45ac-ada0-2e33e84eac60-tmp\") pod \"console-c89c876c6-6smjv\" (UID: \"283648de-3941-45ac-ada0-2e33e84eac60\") " pod="openshift-console/console-c89c876c6-6smjv" Oct 30 21:16:54.517054 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:54.517033 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/283648de-3941-45ac-ada0-2e33e84eac60-console-config\") pod \"console-c89c876c6-6smjv\" (UID: \"283648de-3941-45ac-ada0-2e33e84eac60\") " pod="openshift-console/console-c89c876c6-6smjv" Oct 30 21:16:54.517118 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:54.517082 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/283648de-3941-45ac-ada0-2e33e84eac60-oauth-serving-cert\") pod \"console-c89c876c6-6smjv\" (UID: \"283648de-3941-45ac-ada0-2e33e84eac60\") " pod="openshift-console/console-c89c876c6-6smjv" Oct 30 21:16:54.517215 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:54.517192 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/283648de-3941-45ac-ada0-2e33e84eac60-service-ca\") pod \"console-c89c876c6-6smjv\" (UID: \"283648de-3941-45ac-ada0-2e33e84eac60\") " pod="openshift-console/console-c89c876c6-6smjv" Oct 30 21:16:54.517466 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:54.517449 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/283648de-3941-45ac-ada0-2e33e84eac60-trusted-ca-bundle\") pod \"console-c89c876c6-6smjv\" (UID: \"283648de-3941-45ac-ada0-2e33e84eac60\") " pod="openshift-console/console-c89c876c6-6smjv" Oct 30 21:16:54.518629 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:54.518609 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/283648de-3941-45ac-ada0-2e33e84eac60-console-oauth-config\") pod \"console-c89c876c6-6smjv\" (UID: \"283648de-3941-45ac-ada0-2e33e84eac60\") " pod="openshift-console/console-c89c876c6-6smjv" Oct 30 21:16:54.518941 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:54.518923 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/283648de-3941-45ac-ada0-2e33e84eac60-console-serving-cert\") pod \"console-c89c876c6-6smjv\" (UID: \"283648de-3941-45ac-ada0-2e33e84eac60\") " pod="openshift-console/console-c89c876c6-6smjv" Oct 30 21:16:54.574185 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:54.574154 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b65w\" (UniqueName: \"kubernetes.io/projected/283648de-3941-45ac-ada0-2e33e84eac60-kube-api-access-6b65w\") pod \"console-c89c876c6-6smjv\" (UID: \"283648de-3941-45ac-ada0-2e33e84eac60\") " pod="openshift-console/console-c89c876c6-6smjv" Oct 30 21:16:54.684314 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:54.684234 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c89c876c6-6smjv" Oct 30 21:16:55.051765 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:55.051734 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c89c876c6-6smjv"] Oct 30 21:16:55.059125 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:16:55.059102 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod283648de_3941_45ac_ada0_2e33e84eac60.slice/crio-09f010d13833193c3c74141939049e44a325e0790f60259d6cb2d2a567c08bdf WatchSource:0}: Error finding container 09f010d13833193c3c74141939049e44a325e0790f60259d6cb2d2a567c08bdf: Status 404 returned error can't find the container with id 09f010d13833193c3c74141939049e44a325e0790f60259d6cb2d2a567c08bdf Oct 30 21:16:55.239909 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:55.239870 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c89c876c6-6smjv" event={"ID":"283648de-3941-45ac-ada0-2e33e84eac60","Type":"ContainerStarted","Data":"79cd211812c948ee3cae0a09c3018060d1f62d1508dbd209878345b93793b44c"} Oct 30 21:16:55.240103 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:55.239923 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c89c876c6-6smjv" event={"ID":"283648de-3941-45ac-ada0-2e33e84eac60","Type":"ContainerStarted","Data":"09f010d13833193c3c74141939049e44a325e0790f60259d6cb2d2a567c08bdf"} Oct 30 21:16:55.400318 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:16:55.400213 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-c89c876c6-6smjv" podStartSLOduration=1.4001979 podStartE2EDuration="1.4001979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:55.353224714 +0000 UTC m=+190.907924492" watchObservedRunningTime="2026-10-30 21:16:55.4001979 +0000 UTC m=+190.954897724" Oct 30 21:17:04.684490 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:04.684462 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c89c876c6-6smjv" Oct 30 21:17:04.684916 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:04.684514 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c89c876c6-6smjv" Oct 30 21:17:04.689480 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:04.689459 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c89c876c6-6smjv" Oct 30 21:17:05.274621 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:05.274591 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c89c876c6-6smjv" Oct 30 21:17:05.434587 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:05.434558 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d6bfcf686-w5txk"] Oct 30 21:17:19.682848 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:17:19.682803 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:17:30.453432 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:30.453320 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5d6bfcf686-w5txk" podUID="392b99e3-42e2-4bc9-9421-917b4cb5b83b" containerName="console" containerID="cri-o://94c5c6601e9a7401821d744b503baf6c0d4971cf5f8df1b2d6a2ab5c738ecea3" gracePeriod=15 Oct 30 21:17:30.694262 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:30.694241 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d6bfcf686-w5txk" Oct 30 21:17:30.769059 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:30.769029 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/392b99e3-42e2-4bc9-9421-917b4cb5b83b-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/392b99e3-42e2-4bc9-9421-917b4cb5b83b-console-serving-cert\") pod \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\" (UID: \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\") " Oct 30 21:17:30.769201 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:30.769091 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/392b99e3-42e2-4bc9-9421-917b4cb5b83b-tmp\" (UniqueName: \"kubernetes.io/empty-dir/392b99e3-42e2-4bc9-9421-917b4cb5b83b-tmp\") pod \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\" (UID: \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\") " Oct 30 21:17:30.769201 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:30.769120 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/392b99e3-42e2-4bc9-9421-917b4cb5b83b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/392b99e3-42e2-4bc9-9421-917b4cb5b83b-trusted-ca-bundle\") pod \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\" (UID: \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\") " Oct 30 21:17:30.769201 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:30.769148 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/392b99e3-42e2-4bc9-9421-917b4cb5b83b-console-config\" (UniqueName: \"kubernetes.io/configmap/392b99e3-42e2-4bc9-9421-917b4cb5b83b-console-config\") pod \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\" (UID: \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\") " Oct 30 21:17:30.769201 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:30.769181 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/392b99e3-42e2-4bc9-9421-917b4cb5b83b-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/392b99e3-42e2-4bc9-9421-917b4cb5b83b-console-oauth-config\") pod \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\" (UID: \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\") " Oct 30 21:17:30.769442 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:30.769214 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/392b99e3-42e2-4bc9-9421-917b4cb5b83b-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/392b99e3-42e2-4bc9-9421-917b4cb5b83b-oauth-serving-cert\") pod \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\" (UID: \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\") " Oct 30 21:17:30.769442 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:30.769268 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/392b99e3-42e2-4bc9-9421-917b4cb5b83b-service-ca\" (UniqueName: \"kubernetes.io/configmap/392b99e3-42e2-4bc9-9421-917b4cb5b83b-service-ca\") pod \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\" (UID: \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\") " Oct 30 21:17:30.769442 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:30.769373 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/392b99e3-42e2-4bc9-9421-917b4cb5b83b-kube-api-access-f8jkq\" (UniqueName: \"kubernetes.io/projected/392b99e3-42e2-4bc9-9421-917b4cb5b83b-kube-api-access-f8jkq\") pod \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\" (UID: \"392b99e3-42e2-4bc9-9421-917b4cb5b83b\") " Oct 30 21:17:30.769442 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:30.769398 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/392b99e3-42e2-4bc9-9421-917b4cb5b83b-tmp" pod "392b99e3-42e2-4bc9-9421-917b4cb5b83b" (UID: "392b99e3-42e2-4bc9-9421-917b4cb5b83b"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:30.769623 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:30.769549 4560 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/392b99e3-42e2-4bc9-9421-917b4cb5b83b-tmp\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:17:30.769752 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:30.769717 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/392b99e3-42e2-4bc9-9421-917b4cb5b83b-trusted-ca-bundle" pod "392b99e3-42e2-4bc9-9421-917b4cb5b83b" (UID: "392b99e3-42e2-4bc9-9421-917b4cb5b83b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:17:30.769849 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:30.769818 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/392b99e3-42e2-4bc9-9421-917b4cb5b83b-console-config" pod "392b99e3-42e2-4bc9-9421-917b4cb5b83b" (UID: "392b99e3-42e2-4bc9-9421-917b4cb5b83b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:17:30.769971 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:30.769860 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/392b99e3-42e2-4bc9-9421-917b4cb5b83b-oauth-serving-cert" pod "392b99e3-42e2-4bc9-9421-917b4cb5b83b" (UID: "392b99e3-42e2-4bc9-9421-917b4cb5b83b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:17:30.769971 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:30.769869 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/392b99e3-42e2-4bc9-9421-917b4cb5b83b-service-ca" pod "392b99e3-42e2-4bc9-9421-917b4cb5b83b" (UID: "392b99e3-42e2-4bc9-9421-917b4cb5b83b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:17:30.771304 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:30.771274 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/392b99e3-42e2-4bc9-9421-917b4cb5b83b-console-oauth-config" pod "392b99e3-42e2-4bc9-9421-917b4cb5b83b" (UID: "392b99e3-42e2-4bc9-9421-917b4cb5b83b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:17:30.771451 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:30.771349 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/392b99e3-42e2-4bc9-9421-917b4cb5b83b-console-serving-cert" pod "392b99e3-42e2-4bc9-9421-917b4cb5b83b" (UID: "392b99e3-42e2-4bc9-9421-917b4cb5b83b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:17:30.771451 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:30.771394 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/392b99e3-42e2-4bc9-9421-917b4cb5b83b-kube-api-access-f8jkq" pod "392b99e3-42e2-4bc9-9421-917b4cb5b83b" (UID: "392b99e3-42e2-4bc9-9421-917b4cb5b83b"). InnerVolumeSpecName "kube-api-access-f8jkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:17:30.870521 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:30.870488 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8jkq\" (UniqueName: \"kubernetes.io/projected/392b99e3-42e2-4bc9-9421-917b4cb5b83b-kube-api-access-f8jkq\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:17:30.870521 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:30.870514 4560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/392b99e3-42e2-4bc9-9421-917b4cb5b83b-console-serving-cert\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:17:30.870521 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:30.870524 4560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/392b99e3-42e2-4bc9-9421-917b4cb5b83b-trusted-ca-bundle\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:17:30.870741 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:30.870533 4560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/392b99e3-42e2-4bc9-9421-917b4cb5b83b-console-config\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:17:30.870741 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:30.870542 4560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/392b99e3-42e2-4bc9-9421-917b4cb5b83b-console-oauth-config\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:17:30.870741 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:30.870554 4560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/392b99e3-42e2-4bc9-9421-917b4cb5b83b-oauth-serving-cert\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:17:30.870741 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:30.870565 4560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/392b99e3-42e2-4bc9-9421-917b4cb5b83b-service-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:17:31.353139 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:31.353107 4560 generic.go:356] "Generic (PLEG): container finished" podID="392b99e3-42e2-4bc9-9421-917b4cb5b83b" containerID="94c5c6601e9a7401821d744b503baf6c0d4971cf5f8df1b2d6a2ab5c738ecea3" exitCode=0 Oct 30 21:17:31.353276 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:31.353155 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d6bfcf686-w5txk" event={"ID":"392b99e3-42e2-4bc9-9421-917b4cb5b83b","Type":"ContainerDied","Data":"94c5c6601e9a7401821d744b503baf6c0d4971cf5f8df1b2d6a2ab5c738ecea3"} Oct 30 21:17:31.353276 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:31.353158 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d6bfcf686-w5txk" Oct 30 21:17:31.353276 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:31.353191 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d6bfcf686-w5txk" event={"ID":"392b99e3-42e2-4bc9-9421-917b4cb5b83b","Type":"ContainerDied","Data":"126fbc35959cda6b6157287f30c6373c2c76e4758c33cc52fd84806430519d46"} Oct 30 21:17:31.353276 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:31.353229 4560 scope.go:122] "RemoveContainer" containerID="94c5c6601e9a7401821d744b503baf6c0d4971cf5f8df1b2d6a2ab5c738ecea3" Oct 30 21:17:31.361077 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:31.361058 4560 scope.go:122] "RemoveContainer" containerID="94c5c6601e9a7401821d744b503baf6c0d4971cf5f8df1b2d6a2ab5c738ecea3" Oct 30 21:17:31.361414 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:17:31.361384 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94c5c6601e9a7401821d744b503baf6c0d4971cf5f8df1b2d6a2ab5c738ecea3\": container with ID starting with 94c5c6601e9a7401821d744b503baf6c0d4971cf5f8df1b2d6a2ab5c738ecea3 not found: ID does not exist" containerID="94c5c6601e9a7401821d744b503baf6c0d4971cf5f8df1b2d6a2ab5c738ecea3" Oct 30 21:17:31.361515 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:31.361427 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94c5c6601e9a7401821d744b503baf6c0d4971cf5f8df1b2d6a2ab5c738ecea3"} err="failed to get container status \"94c5c6601e9a7401821d744b503baf6c0d4971cf5f8df1b2d6a2ab5c738ecea3\": rpc error: code = NotFound desc = could not find container \"94c5c6601e9a7401821d744b503baf6c0d4971cf5f8df1b2d6a2ab5c738ecea3\": container with ID starting with 94c5c6601e9a7401821d744b503baf6c0d4971cf5f8df1b2d6a2ab5c738ecea3 not found: ID does not exist" Oct 30 21:17:31.507580 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:31.507547 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d6bfcf686-w5txk"] Oct 30 21:17:31.554289 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:31.554252 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d6bfcf686-w5txk"] Oct 30 21:17:32.572657 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:17:32.572623 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="392b99e3-42e2-4bc9-9421-917b4cb5b83b" path="/var/lib/kubelet/pods/392b99e3-42e2-4bc9-9421-917b4cb5b83b/volumes" Oct 30 21:17:49.682490 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:17:49.682450 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:18:19.683413 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:18:19.683375 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:18:43.775256 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:43.775221 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn"] Oct 30 21:18:43.775639 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:43.775478 4560 cpu_manager.go:414] "containerMap: removing container" podUID="392b99e3-42e2-4bc9-9421-917b4cb5b83b" containerName="console" Oct 30 21:18:43.775639 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:43.775493 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="392b99e3-42e2-4bc9-9421-917b4cb5b83b" containerName="console" Oct 30 21:18:43.775639 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:43.775564 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="392b99e3-42e2-4bc9-9421-917b4cb5b83b" containerName="console" Oct 30 21:18:43.777489 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:43.777473 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn" Oct 30 21:18:43.827332 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:43.827298 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 21:18:43.827489 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:43.827381 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 21:18:43.827489 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:43.827410 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 21:18:43.827489 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:43.827437 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 21:18:43.827696 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:43.827613 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 21:18:43.873012 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:43.872983 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0486dfec-28d9-430e-8816-92f8869a7bf9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-bmwnn\" (UID: \"0486dfec-28d9-430e-8816-92f8869a7bf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn" Oct 30 21:18:43.873143 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:43.873028 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0486dfec-28d9-430e-8816-92f8869a7bf9-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-bmwnn\" (UID: \"0486dfec-28d9-430e-8816-92f8869a7bf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn" Oct 30 21:18:43.873143 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:43.873110 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0486dfec-28d9-430e-8816-92f8869a7bf9-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-bmwnn\" (UID: \"0486dfec-28d9-430e-8816-92f8869a7bf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn" Oct 30 21:18:43.873217 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:43.873187 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0486dfec-28d9-430e-8816-92f8869a7bf9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-bmwnn\" (UID: \"0486dfec-28d9-430e-8816-92f8869a7bf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn" Oct 30 21:18:43.873250 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:43.873215 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55stx\" (UniqueName: \"kubernetes.io/projected/0486dfec-28d9-430e-8816-92f8869a7bf9-kube-api-access-55stx\") pod \"istiod-openshift-gateway-7876bfcb74-bmwnn\" (UID: \"0486dfec-28d9-430e-8816-92f8869a7bf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn" Oct 30 21:18:43.873250 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:43.873243 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0486dfec-28d9-430e-8816-92f8869a7bf9-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-bmwnn\" (UID: \"0486dfec-28d9-430e-8816-92f8869a7bf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn" Oct 30 21:18:43.873319 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:43.873260 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0486dfec-28d9-430e-8816-92f8869a7bf9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-bmwnn\" (UID: \"0486dfec-28d9-430e-8816-92f8869a7bf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn" Oct 30 21:18:43.874305 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:43.874291 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 21:18:43.874383 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:43.874366 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-vmmk5\"" Oct 30 21:18:43.875202 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:43.875186 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn"] Oct 30 21:18:43.974577 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:43.974552 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0486dfec-28d9-430e-8816-92f8869a7bf9-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-bmwnn\" (UID: \"0486dfec-28d9-430e-8816-92f8869a7bf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn" Oct 30 21:18:43.974577 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:43.974579 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0486dfec-28d9-430e-8816-92f8869a7bf9-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-bmwnn\" (UID: \"0486dfec-28d9-430e-8816-92f8869a7bf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn" Oct 30 21:18:43.974757 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:43.974613 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0486dfec-28d9-430e-8816-92f8869a7bf9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-bmwnn\" (UID: \"0486dfec-28d9-430e-8816-92f8869a7bf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn" Oct 30 21:18:43.974757 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:43.974741 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55stx\" (UniqueName: \"kubernetes.io/projected/0486dfec-28d9-430e-8816-92f8869a7bf9-kube-api-access-55stx\") pod \"istiod-openshift-gateway-7876bfcb74-bmwnn\" (UID: \"0486dfec-28d9-430e-8816-92f8869a7bf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn" Oct 30 21:18:43.974872 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:43.974800 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0486dfec-28d9-430e-8816-92f8869a7bf9-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-bmwnn\" (UID: \"0486dfec-28d9-430e-8816-92f8869a7bf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn" Oct 30 21:18:43.974872 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:43.974830 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0486dfec-28d9-430e-8816-92f8869a7bf9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-bmwnn\" (UID: \"0486dfec-28d9-430e-8816-92f8869a7bf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn" Oct 30 21:18:43.974969 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:43.974865 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0486dfec-28d9-430e-8816-92f8869a7bf9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-bmwnn\" (UID: \"0486dfec-28d9-430e-8816-92f8869a7bf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn" Oct 30 21:18:43.975415 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:43.975378 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0486dfec-28d9-430e-8816-92f8869a7bf9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-bmwnn\" (UID: \"0486dfec-28d9-430e-8816-92f8869a7bf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn" Oct 30 21:18:43.976958 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:43.976939 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0486dfec-28d9-430e-8816-92f8869a7bf9-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-bmwnn\" (UID: \"0486dfec-28d9-430e-8816-92f8869a7bf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn" Oct 30 21:18:43.977040 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:43.976981 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0486dfec-28d9-430e-8816-92f8869a7bf9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-bmwnn\" (UID: \"0486dfec-28d9-430e-8816-92f8869a7bf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn" Oct 30 21:18:43.977143 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:43.977128 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0486dfec-28d9-430e-8816-92f8869a7bf9-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-bmwnn\" (UID: \"0486dfec-28d9-430e-8816-92f8869a7bf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn" Oct 30 21:18:43.977179 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:43.977130 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0486dfec-28d9-430e-8816-92f8869a7bf9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-bmwnn\" (UID: \"0486dfec-28d9-430e-8816-92f8869a7bf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn" Oct 30 21:18:44.032641 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:44.032584 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0486dfec-28d9-430e-8816-92f8869a7bf9-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-bmwnn\" (UID: \"0486dfec-28d9-430e-8816-92f8869a7bf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn" Oct 30 21:18:44.032853 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:44.032833 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55stx\" (UniqueName: \"kubernetes.io/projected/0486dfec-28d9-430e-8816-92f8869a7bf9-kube-api-access-55stx\") pod \"istiod-openshift-gateway-7876bfcb74-bmwnn\" (UID: \"0486dfec-28d9-430e-8816-92f8869a7bf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn" Oct 30 21:18:44.087880 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:44.087857 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn" Oct 30 21:18:44.508286 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:44.508252 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn"] Oct 30 21:18:44.514655 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:18:44.514627 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0486dfec_28d9_430e_8816_92f8869a7bf9.slice/crio-619df02ee468ce5bd45f6b15feebc79f1621ef1c06a1922e44e5fe90d04b505e WatchSource:0}: Error finding container 619df02ee468ce5bd45f6b15feebc79f1621ef1c06a1922e44e5fe90d04b505e: Status 404 returned error can't find the container with id 619df02ee468ce5bd45f6b15feebc79f1621ef1c06a1922e44e5fe90d04b505e Oct 30 21:18:44.580272 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:44.580245 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn" event={"ID":"0486dfec-28d9-430e-8816-92f8869a7bf9","Type":"ContainerStarted","Data":"619df02ee468ce5bd45f6b15feebc79f1621ef1c06a1922e44e5fe90d04b505e"} Oct 30 21:18:48.478316 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:48.478287 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:18:48.478571 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:48.478364 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:18:48.593618 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:48.593584 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn" event={"ID":"0486dfec-28d9-430e-8816-92f8869a7bf9","Type":"ContainerStarted","Data":"5add80301a6ff4b6e35734b3162b0c948dc88c42349d519ce462ad865f3bcc04"} Oct 30 21:18:48.593798 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:48.593698 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn" Oct 30 21:18:48.794041 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:48.793952 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn" podStartSLOduration=1.832957456 podStartE2EDuration="5.793940303s" totalImagesPullingTime="3.960982847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:48.746966217 +0000 UTC m=+304.301666005" watchObservedRunningTime="2026-10-30 21:18:48.793940303 +0000 UTC m=+304.348640079" Oct 30 21:18:49.598241 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:49.598208 4560 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-bmwnn container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:18:49.598638 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:49.598268 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn" podUID="0486dfec-28d9-430e-8816-92f8869a7bf9" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:18:49.683508 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:18:49.683470 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:18:52.597359 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:18:52.597323 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn" Oct 30 21:19:19.683122 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:19:19.683043 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:19:39.265353 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:19:39.265311 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-4xmb4"] Oct 30 21:19:39.268313 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:19:39.268297 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-4xmb4" Oct 30 21:19:39.373674 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:19:39.373638 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/225a51cb-15f7-41c3-aea1-f53d4728f77e-manifests\") pod \"opendatahub-operator-controller-manager-695f8457f8-4xmb4\" (UID: \"225a51cb-15f7-41c3-aea1-f53d4728f77e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-4xmb4" Oct 30 21:19:39.373674 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:19:39.373675 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/225a51cb-15f7-41c3-aea1-f53d4728f77e-tmp\") pod \"opendatahub-operator-controller-manager-695f8457f8-4xmb4\" (UID: \"225a51cb-15f7-41c3-aea1-f53d4728f77e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-4xmb4" Oct 30 21:19:39.373859 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:19:39.373696 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/225a51cb-15f7-41c3-aea1-f53d4728f77e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-695f8457f8-4xmb4\" (UID: \"225a51cb-15f7-41c3-aea1-f53d4728f77e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-4xmb4" Oct 30 21:19:39.373859 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:19:39.373759 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/225a51cb-15f7-41c3-aea1-f53d4728f77e-webhook-cert\") pod \"opendatahub-operator-controller-manager-695f8457f8-4xmb4\" (UID: \"225a51cb-15f7-41c3-aea1-f53d4728f77e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-4xmb4" Oct 30 21:19:39.373859 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:19:39.373816 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/225a51cb-15f7-41c3-aea1-f53d4728f77e-metrics-cert\") pod \"opendatahub-operator-controller-manager-695f8457f8-4xmb4\" (UID: \"225a51cb-15f7-41c3-aea1-f53d4728f77e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-4xmb4" Oct 30 21:19:39.373958 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:19:39.373901 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhzp6\" (UniqueName: \"kubernetes.io/projected/225a51cb-15f7-41c3-aea1-f53d4728f77e-kube-api-access-lhzp6\") pod \"opendatahub-operator-controller-manager-695f8457f8-4xmb4\" (UID: \"225a51cb-15f7-41c3-aea1-f53d4728f77e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-4xmb4" Oct 30 21:19:39.392197 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:19:39.392176 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 21:19:39.392255 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:19:39.392220 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 21:19:39.392314 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:19:39.392293 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 21:19:39.392415 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:19:39.392378 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 21:19:39.392486 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:19:39.392422 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-5lv4p\"" Oct 30 21:19:39.392486 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:19:39.392446 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 21:19:39.474783 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:19:39.474744 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/225a51cb-15f7-41c3-aea1-f53d4728f77e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-695f8457f8-4xmb4\" (UID: \"225a51cb-15f7-41c3-aea1-f53d4728f77e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-4xmb4" Oct 30 21:19:39.474783 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:19:39.474783 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/225a51cb-15f7-41c3-aea1-f53d4728f77e-webhook-cert\") pod \"opendatahub-operator-controller-manager-695f8457f8-4xmb4\" (UID: \"225a51cb-15f7-41c3-aea1-f53d4728f77e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-4xmb4" Oct 30 21:19:39.475052 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:19:39.474802 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/225a51cb-15f7-41c3-aea1-f53d4728f77e-metrics-cert\") pod \"opendatahub-operator-controller-manager-695f8457f8-4xmb4\" (UID: \"225a51cb-15f7-41c3-aea1-f53d4728f77e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-4xmb4" Oct 30 21:19:39.475052 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:19:39.474840 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhzp6\" (UniqueName: \"kubernetes.io/projected/225a51cb-15f7-41c3-aea1-f53d4728f77e-kube-api-access-lhzp6\") pod \"opendatahub-operator-controller-manager-695f8457f8-4xmb4\" (UID: \"225a51cb-15f7-41c3-aea1-f53d4728f77e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-4xmb4" Oct 30 21:19:39.475052 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:19:39.474883 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/225a51cb-15f7-41c3-aea1-f53d4728f77e-manifests\") pod \"opendatahub-operator-controller-manager-695f8457f8-4xmb4\" (UID: \"225a51cb-15f7-41c3-aea1-f53d4728f77e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-4xmb4" Oct 30 21:19:39.475052 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:19:39.474915 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/225a51cb-15f7-41c3-aea1-f53d4728f77e-tmp\") pod \"opendatahub-operator-controller-manager-695f8457f8-4xmb4\" (UID: \"225a51cb-15f7-41c3-aea1-f53d4728f77e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-4xmb4" Oct 30 21:19:39.475439 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:19:39.475411 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/225a51cb-15f7-41c3-aea1-f53d4728f77e-manifests\") pod \"opendatahub-operator-controller-manager-695f8457f8-4xmb4\" (UID: \"225a51cb-15f7-41c3-aea1-f53d4728f77e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-4xmb4" Oct 30 21:19:39.477259 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:19:39.477236 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/225a51cb-15f7-41c3-aea1-f53d4728f77e-tmp\") pod \"opendatahub-operator-controller-manager-695f8457f8-4xmb4\" (UID: \"225a51cb-15f7-41c3-aea1-f53d4728f77e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-4xmb4" Oct 30 21:19:39.477486 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:19:39.477466 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/225a51cb-15f7-41c3-aea1-f53d4728f77e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-695f8457f8-4xmb4\" (UID: \"225a51cb-15f7-41c3-aea1-f53d4728f77e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-4xmb4" Oct 30 21:19:39.477581 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:19:39.477541 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/225a51cb-15f7-41c3-aea1-f53d4728f77e-webhook-cert\") pod \"opendatahub-operator-controller-manager-695f8457f8-4xmb4\" (UID: \"225a51cb-15f7-41c3-aea1-f53d4728f77e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-4xmb4" Oct 30 21:19:39.477633 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:19:39.477576 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/225a51cb-15f7-41c3-aea1-f53d4728f77e-metrics-cert\") pod \"opendatahub-operator-controller-manager-695f8457f8-4xmb4\" (UID: \"225a51cb-15f7-41c3-aea1-f53d4728f77e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-4xmb4" Oct 30 21:19:39.496737 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:19:39.496708 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-4xmb4"] Oct 30 21:19:39.533201 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:19:39.533135 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhzp6\" (UniqueName: \"kubernetes.io/projected/225a51cb-15f7-41c3-aea1-f53d4728f77e-kube-api-access-lhzp6\") pod \"opendatahub-operator-controller-manager-695f8457f8-4xmb4\" (UID: \"225a51cb-15f7-41c3-aea1-f53d4728f77e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-4xmb4" Oct 30 21:19:39.583221 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:19:39.583183 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-4xmb4" Oct 30 21:19:40.150022 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:19:40.149991 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-4xmb4"] Oct 30 21:19:40.157053 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:19:40.157022 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod225a51cb_15f7_41c3_aea1_f53d4728f77e.slice/crio-8bad7c343141043c9c4b80b150da9e9608ea10c3ec9d9feb795d34c20f488f93 WatchSource:0}: Error finding container 8bad7c343141043c9c4b80b150da9e9608ea10c3ec9d9feb795d34c20f488f93: Status 404 returned error can't find the container with id 8bad7c343141043c9c4b80b150da9e9608ea10c3ec9d9feb795d34c20f488f93 Oct 30 21:19:40.158626 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:19:40.158609 4560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:19:40.751679 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:19:40.751636 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-4xmb4" event={"ID":"225a51cb-15f7-41c3-aea1-f53d4728f77e","Type":"ContainerStarted","Data":"8bad7c343141043c9c4b80b150da9e9608ea10c3ec9d9feb795d34c20f488f93"} Oct 30 21:19:43.763631 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:19:43.763598 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-4xmb4" event={"ID":"225a51cb-15f7-41c3-aea1-f53d4728f77e","Type":"ContainerStarted","Data":"622ff8929cf02dffafe4a2b18560f356b920d3a1c2cfd5f5ff52be861ed79cb0"} Oct 30 21:19:44.769114 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:19:44.769075 4560 generic.go:356] "Generic (PLEG): container finished" podID="225a51cb-15f7-41c3-aea1-f53d4728f77e" containerID="622ff8929cf02dffafe4a2b18560f356b920d3a1c2cfd5f5ff52be861ed79cb0" exitCode=0 Oct 30 21:19:44.769514 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:19:44.769159 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-4xmb4" event={"ID":"225a51cb-15f7-41c3-aea1-f53d4728f77e","Type":"ContainerDied","Data":"622ff8929cf02dffafe4a2b18560f356b920d3a1c2cfd5f5ff52be861ed79cb0"} Oct 30 21:19:45.774323 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:19:45.774288 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-4xmb4" event={"ID":"225a51cb-15f7-41c3-aea1-f53d4728f77e","Type":"ContainerStarted","Data":"3b18772988f7454afc64fd70f0758fee398ffb57bdca9f9a0867dce63a0a9e57"} Oct 30 21:19:45.774717 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:19:45.774431 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-4xmb4" Oct 30 21:19:45.969470 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:19:45.969374 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-4xmb4" podStartSLOduration=2.075414261 podStartE2EDuration="6.969359727s" totalImagesPullingTime="4.408572319s" totalInitContainerRuntime="485.373147ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:45.967793662 +0000 UTC m=+361.522493439" watchObservedRunningTime="2026-10-30 21:19:45.969359727 +0000 UTC m=+361.524059498" Oct 30 21:19:49.682921 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:19:49.682884 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:19:56.779459 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:19:56.779421 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-695f8457f8-4xmb4" Oct 30 21:20:10.073132 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:20:10.073095 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6645f5c86b-xmshw"] Oct 30 21:20:10.076529 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:20:10.076507 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6645f5c86b-xmshw" Oct 30 21:20:10.101196 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:20:10.101163 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkvwv\" (UniqueName: \"kubernetes.io/projected/b7f14427-2d3d-4a1d-a919-2842cded93b0-kube-api-access-mkvwv\") pod \"kube-auth-proxy-6645f5c86b-xmshw\" (UID: \"b7f14427-2d3d-4a1d-a919-2842cded93b0\") " pod="openshift-ingress/kube-auth-proxy-6645f5c86b-xmshw" Oct 30 21:20:10.101391 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:20:10.101220 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7f14427-2d3d-4a1d-a919-2842cded93b0-tls-certs\") pod \"kube-auth-proxy-6645f5c86b-xmshw\" (UID: \"b7f14427-2d3d-4a1d-a919-2842cded93b0\") " pod="openshift-ingress/kube-auth-proxy-6645f5c86b-xmshw" Oct 30 21:20:10.101391 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:20:10.101287 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7f14427-2d3d-4a1d-a919-2842cded93b0-tmp\") pod \"kube-auth-proxy-6645f5c86b-xmshw\" (UID: \"b7f14427-2d3d-4a1d-a919-2842cded93b0\") " pod="openshift-ingress/kube-auth-proxy-6645f5c86b-xmshw" Oct 30 21:20:10.172764 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:20:10.172727 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 21:20:10.172956 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:20:10.172808 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-pwtw7\"" Oct 30 21:20:10.172956 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:20:10.172865 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 21:20:10.174166 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:20:10.174139 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6645f5c86b-xmshw"] Oct 30 21:20:10.202448 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:20:10.202412 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7f14427-2d3d-4a1d-a919-2842cded93b0-tmp\") pod \"kube-auth-proxy-6645f5c86b-xmshw\" (UID: \"b7f14427-2d3d-4a1d-a919-2842cded93b0\") " pod="openshift-ingress/kube-auth-proxy-6645f5c86b-xmshw" Oct 30 21:20:10.202593 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:20:10.202472 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkvwv\" (UniqueName: \"kubernetes.io/projected/b7f14427-2d3d-4a1d-a919-2842cded93b0-kube-api-access-mkvwv\") pod \"kube-auth-proxy-6645f5c86b-xmshw\" (UID: \"b7f14427-2d3d-4a1d-a919-2842cded93b0\") " pod="openshift-ingress/kube-auth-proxy-6645f5c86b-xmshw" Oct 30 21:20:10.202593 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:20:10.202512 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7f14427-2d3d-4a1d-a919-2842cded93b0-tls-certs\") pod \"kube-auth-proxy-6645f5c86b-xmshw\" (UID: \"b7f14427-2d3d-4a1d-a919-2842cded93b0\") " pod="openshift-ingress/kube-auth-proxy-6645f5c86b-xmshw" Oct 30 21:20:10.204702 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:20:10.204680 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7f14427-2d3d-4a1d-a919-2842cded93b0-tmp\") pod \"kube-auth-proxy-6645f5c86b-xmshw\" (UID: \"b7f14427-2d3d-4a1d-a919-2842cded93b0\") " pod="openshift-ingress/kube-auth-proxy-6645f5c86b-xmshw" Oct 30 21:20:10.204849 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:20:10.204831 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7f14427-2d3d-4a1d-a919-2842cded93b0-tls-certs\") pod \"kube-auth-proxy-6645f5c86b-xmshw\" (UID: \"b7f14427-2d3d-4a1d-a919-2842cded93b0\") " pod="openshift-ingress/kube-auth-proxy-6645f5c86b-xmshw" Oct 30 21:20:10.261774 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:20:10.261744 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkvwv\" (UniqueName: \"kubernetes.io/projected/b7f14427-2d3d-4a1d-a919-2842cded93b0-kube-api-access-mkvwv\") pod \"kube-auth-proxy-6645f5c86b-xmshw\" (UID: \"b7f14427-2d3d-4a1d-a919-2842cded93b0\") " pod="openshift-ingress/kube-auth-proxy-6645f5c86b-xmshw" Oct 30 21:20:10.387858 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:20:10.387772 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6645f5c86b-xmshw" Oct 30 21:20:10.788930 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:20:10.788892 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6645f5c86b-xmshw"] Oct 30 21:20:10.796173 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:20:10.796145 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7f14427_2d3d_4a1d_a919_2842cded93b0.slice/crio-e3ce3a4030b38233691df6a5b7a558054bd6426035fa9159a78d2984a22af541 WatchSource:0}: Error finding container e3ce3a4030b38233691df6a5b7a558054bd6426035fa9159a78d2984a22af541: Status 404 returned error can't find the container with id e3ce3a4030b38233691df6a5b7a558054bd6426035fa9159a78d2984a22af541 Oct 30 21:20:10.856071 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:20:10.856040 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6645f5c86b-xmshw" event={"ID":"b7f14427-2d3d-4a1d-a919-2842cded93b0","Type":"ContainerStarted","Data":"e3ce3a4030b38233691df6a5b7a558054bd6426035fa9159a78d2984a22af541"} Oct 30 21:20:14.872493 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:20:14.872448 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6645f5c86b-xmshw" event={"ID":"b7f14427-2d3d-4a1d-a919-2842cded93b0","Type":"ContainerStarted","Data":"2c7838b3edc2024302291186e5b8d767d3941b7341d95089d33050366199b6a8"} Oct 30 21:20:14.982704 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:20:14.982647 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6645f5c86b-xmshw" podStartSLOduration=1.855935025 podStartE2EDuration="4.982634074s" totalImagesPullingTime="3.126699049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:14.982425926 +0000 UTC m=+390.537125702" watchObservedRunningTime="2026-10-30 21:20:14.982634074 +0000 UTC m=+390.537333851" Oct 30 21:20:19.684633 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:20:19.684597 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:20:49.683376 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:20:49.683319 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:21:19.682852 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:21:19.682818 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:21:24.070331 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:24.070289 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngwjn"] Oct 30 21:21:24.074912 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:24.074888 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngwjn" Oct 30 21:21:24.170693 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:24.170654 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngwjn"] Oct 30 21:21:24.266147 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:24.266117 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d9bb2bf8-3fce-443b-a561-ed34cc9e9c61-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngwjn\" (UID: \"d9bb2bf8-3fce-443b-a561-ed34cc9e9c61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngwjn" Oct 30 21:21:24.266322 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:24.266158 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9bb2bf8-3fce-443b-a561-ed34cc9e9c61-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngwjn\" (UID: \"d9bb2bf8-3fce-443b-a561-ed34cc9e9c61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngwjn" Oct 30 21:21:24.266322 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:24.266177 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9bb2bf8-3fce-443b-a561-ed34cc9e9c61-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngwjn\" (UID: \"d9bb2bf8-3fce-443b-a561-ed34cc9e9c61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngwjn" Oct 30 21:21:24.266322 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:24.266279 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wb45\" (UniqueName: \"kubernetes.io/projected/d9bb2bf8-3fce-443b-a561-ed34cc9e9c61-kube-api-access-9wb45\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngwjn\" (UID: \"d9bb2bf8-3fce-443b-a561-ed34cc9e9c61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngwjn" Oct 30 21:21:24.266500 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:24.266332 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d9bb2bf8-3fce-443b-a561-ed34cc9e9c61-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngwjn\" (UID: \"d9bb2bf8-3fce-443b-a561-ed34cc9e9c61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngwjn" Oct 30 21:21:24.266500 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:24.266383 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d9bb2bf8-3fce-443b-a561-ed34cc9e9c61-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngwjn\" (UID: \"d9bb2bf8-3fce-443b-a561-ed34cc9e9c61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngwjn" Oct 30 21:21:24.266500 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:24.266418 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d9bb2bf8-3fce-443b-a561-ed34cc9e9c61-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngwjn\" (UID: \"d9bb2bf8-3fce-443b-a561-ed34cc9e9c61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngwjn" Oct 30 21:21:24.367684 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:24.367600 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d9bb2bf8-3fce-443b-a561-ed34cc9e9c61-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngwjn\" (UID: \"d9bb2bf8-3fce-443b-a561-ed34cc9e9c61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngwjn" Oct 30 21:21:24.367684 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:24.367653 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9bb2bf8-3fce-443b-a561-ed34cc9e9c61-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngwjn\" (UID: \"d9bb2bf8-3fce-443b-a561-ed34cc9e9c61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngwjn" Oct 30 21:21:24.367684 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:24.367678 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9bb2bf8-3fce-443b-a561-ed34cc9e9c61-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngwjn\" (UID: \"d9bb2bf8-3fce-443b-a561-ed34cc9e9c61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngwjn" Oct 30 21:21:24.367942 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:24.367711 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wb45\" (UniqueName: \"kubernetes.io/projected/d9bb2bf8-3fce-443b-a561-ed34cc9e9c61-kube-api-access-9wb45\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngwjn\" (UID: \"d9bb2bf8-3fce-443b-a561-ed34cc9e9c61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngwjn" Oct 30 21:21:24.367942 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:24.367833 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d9bb2bf8-3fce-443b-a561-ed34cc9e9c61-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngwjn\" (UID: \"d9bb2bf8-3fce-443b-a561-ed34cc9e9c61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngwjn" Oct 30 21:21:24.367942 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:24.367893 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d9bb2bf8-3fce-443b-a561-ed34cc9e9c61-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngwjn\" (UID: \"d9bb2bf8-3fce-443b-a561-ed34cc9e9c61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngwjn" Oct 30 21:21:24.368101 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:24.367970 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d9bb2bf8-3fce-443b-a561-ed34cc9e9c61-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngwjn\" (UID: \"d9bb2bf8-3fce-443b-a561-ed34cc9e9c61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngwjn" Oct 30 21:21:24.368785 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:24.368756 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d9bb2bf8-3fce-443b-a561-ed34cc9e9c61-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngwjn\" (UID: \"d9bb2bf8-3fce-443b-a561-ed34cc9e9c61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngwjn" Oct 30 21:21:24.369965 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:24.369941 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9bb2bf8-3fce-443b-a561-ed34cc9e9c61-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngwjn\" (UID: \"d9bb2bf8-3fce-443b-a561-ed34cc9e9c61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngwjn" Oct 30 21:21:24.370071 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:24.370016 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d9bb2bf8-3fce-443b-a561-ed34cc9e9c61-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngwjn\" (UID: \"d9bb2bf8-3fce-443b-a561-ed34cc9e9c61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngwjn" Oct 30 21:21:24.370130 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:24.370114 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d9bb2bf8-3fce-443b-a561-ed34cc9e9c61-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngwjn\" (UID: \"d9bb2bf8-3fce-443b-a561-ed34cc9e9c61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngwjn" Oct 30 21:21:24.370185 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:24.370173 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d9bb2bf8-3fce-443b-a561-ed34cc9e9c61-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngwjn\" (UID: \"d9bb2bf8-3fce-443b-a561-ed34cc9e9c61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngwjn" Oct 30 21:21:24.425585 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:24.425560 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9bb2bf8-3fce-443b-a561-ed34cc9e9c61-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngwjn\" (UID: \"d9bb2bf8-3fce-443b-a561-ed34cc9e9c61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngwjn" Oct 30 21:21:24.426065 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:24.426043 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wb45\" (UniqueName: \"kubernetes.io/projected/d9bb2bf8-3fce-443b-a561-ed34cc9e9c61-kube-api-access-9wb45\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngwjn\" (UID: \"d9bb2bf8-3fce-443b-a561-ed34cc9e9c61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngwjn" Oct 30 21:21:24.685276 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:24.685208 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngwjn" Oct 30 21:21:25.171692 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:25.171662 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngwjn"] Oct 30 21:21:25.178498 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:21:25.178470 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9bb2bf8_3fce_443b_a561_ed34cc9e9c61.slice/crio-12a5ba84e432fd4c82c50fd0f927324d406652d5f8ee6b173770a00ace481457 WatchSource:0}: Error finding container 12a5ba84e432fd4c82c50fd0f927324d406652d5f8ee6b173770a00ace481457: Status 404 returned error can't find the container with id 12a5ba84e432fd4c82c50fd0f927324d406652d5f8ee6b173770a00ace481457 Oct 30 21:21:25.180462 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:25.180429 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:21:25.180535 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:25.180490 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:21:26.107329 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:26.107293 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngwjn" event={"ID":"d9bb2bf8-3fce-443b-a561-ed34cc9e9c61","Type":"ContainerStarted","Data":"e2591500503d0ac8be0f96df4939f1dd7584cebc8ec804cc4e8752d0649ed595"} Oct 30 21:21:26.107538 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:26.107369 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngwjn" event={"ID":"d9bb2bf8-3fce-443b-a561-ed34cc9e9c61","Type":"ContainerStarted","Data":"12a5ba84e432fd4c82c50fd0f927324d406652d5f8ee6b173770a00ace481457"} Oct 30 21:21:26.107538 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:26.107494 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngwjn" Oct 30 21:21:26.109101 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:26.109081 4560 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-ngwjn container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:21:26.109198 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:26.109121 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngwjn" podUID="d9bb2bf8-3fce-443b-a561-ed34cc9e9c61" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:21:27.112265 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:27.112239 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngwjn" Oct 30 21:21:27.321333 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:27.321268 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngwjn" podStartSLOduration=3.3212526430000002 podStartE2EDuration="3.321252643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:26.32227441 +0000 UTC m=+461.876974187" watchObservedRunningTime="2026-10-30 21:21:27.321252643 +0000 UTC m=+462.875952420" Oct 30 21:21:27.322528 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:27.322502 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn"] Oct 30 21:21:27.322714 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:27.322693 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn" podUID="0486dfec-28d9-430e-8816-92f8869a7bf9" containerName="discovery" containerID="cri-o://5add80301a6ff4b6e35734b3162b0c948dc88c42349d519ce462ad865f3bcc04" gracePeriod=30 Oct 30 21:21:27.570143 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:27.570119 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn" Oct 30 21:21:27.595957 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:27.595927 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0486dfec-28d9-430e-8816-92f8869a7bf9-kube-api-access-55stx\" (UniqueName: \"kubernetes.io/projected/0486dfec-28d9-430e-8816-92f8869a7bf9-kube-api-access-55stx\") pod \"0486dfec-28d9-430e-8816-92f8869a7bf9\" (UID: \"0486dfec-28d9-430e-8816-92f8869a7bf9\") " Oct 30 21:21:27.596123 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:27.595976 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0486dfec-28d9-430e-8816-92f8869a7bf9-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0486dfec-28d9-430e-8816-92f8869a7bf9-local-certs\") pod \"0486dfec-28d9-430e-8816-92f8869a7bf9\" (UID: \"0486dfec-28d9-430e-8816-92f8869a7bf9\") " Oct 30 21:21:27.596123 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:27.596027 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0486dfec-28d9-430e-8816-92f8869a7bf9-istio-token\" (UniqueName: \"kubernetes.io/projected/0486dfec-28d9-430e-8816-92f8869a7bf9-istio-token\") pod \"0486dfec-28d9-430e-8816-92f8869a7bf9\" (UID: \"0486dfec-28d9-430e-8816-92f8869a7bf9\") " Oct 30 21:21:27.596123 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:27.596063 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0486dfec-28d9-430e-8816-92f8869a7bf9-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0486dfec-28d9-430e-8816-92f8869a7bf9-istio-kubeconfig\") pod \"0486dfec-28d9-430e-8816-92f8869a7bf9\" (UID: \"0486dfec-28d9-430e-8816-92f8869a7bf9\") " Oct 30 21:21:27.596123 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:27.596097 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0486dfec-28d9-430e-8816-92f8869a7bf9-cacerts\" (UniqueName: \"kubernetes.io/secret/0486dfec-28d9-430e-8816-92f8869a7bf9-cacerts\") pod \"0486dfec-28d9-430e-8816-92f8869a7bf9\" (UID: \"0486dfec-28d9-430e-8816-92f8869a7bf9\") " Oct 30 21:21:27.596361 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:27.596222 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0486dfec-28d9-430e-8816-92f8869a7bf9-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0486dfec-28d9-430e-8816-92f8869a7bf9-istio-csr-ca-configmap\") pod \"0486dfec-28d9-430e-8816-92f8869a7bf9\" (UID: \"0486dfec-28d9-430e-8816-92f8869a7bf9\") " Oct 30 21:21:27.596361 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:27.596320 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0486dfec-28d9-430e-8816-92f8869a7bf9-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0486dfec-28d9-430e-8816-92f8869a7bf9-istio-csr-dns-cert\") pod \"0486dfec-28d9-430e-8816-92f8869a7bf9\" (UID: \"0486dfec-28d9-430e-8816-92f8869a7bf9\") " Oct 30 21:21:27.596614 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:27.596584 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0486dfec-28d9-430e-8816-92f8869a7bf9-istio-csr-ca-configmap" pod "0486dfec-28d9-430e-8816-92f8869a7bf9" (UID: "0486dfec-28d9-430e-8816-92f8869a7bf9"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:21:27.598472 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:27.598444 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0486dfec-28d9-430e-8816-92f8869a7bf9-istio-kubeconfig" pod "0486dfec-28d9-430e-8816-92f8869a7bf9" (UID: "0486dfec-28d9-430e-8816-92f8869a7bf9"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:21:27.598472 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:27.598461 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0486dfec-28d9-430e-8816-92f8869a7bf9-kube-api-access-55stx" pod "0486dfec-28d9-430e-8816-92f8869a7bf9" (UID: "0486dfec-28d9-430e-8816-92f8869a7bf9"). InnerVolumeSpecName "kube-api-access-55stx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:21:27.598626 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:27.598566 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0486dfec-28d9-430e-8816-92f8869a7bf9-cacerts" pod "0486dfec-28d9-430e-8816-92f8869a7bf9" (UID: "0486dfec-28d9-430e-8816-92f8869a7bf9"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:21:27.598679 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:27.598633 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0486dfec-28d9-430e-8816-92f8869a7bf9-local-certs" pod "0486dfec-28d9-430e-8816-92f8869a7bf9" (UID: "0486dfec-28d9-430e-8816-92f8869a7bf9"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:21:27.598730 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:27.598676 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0486dfec-28d9-430e-8816-92f8869a7bf9-istio-token" pod "0486dfec-28d9-430e-8816-92f8869a7bf9" (UID: "0486dfec-28d9-430e-8816-92f8869a7bf9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:21:27.598832 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:27.598817 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0486dfec-28d9-430e-8816-92f8869a7bf9-istio-csr-dns-cert" pod "0486dfec-28d9-430e-8816-92f8869a7bf9" (UID: "0486dfec-28d9-430e-8816-92f8869a7bf9"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:21:27.697813 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:27.697742 4560 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0486dfec-28d9-430e-8816-92f8869a7bf9-istio-kubeconfig\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:21:27.697813 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:27.697768 4560 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0486dfec-28d9-430e-8816-92f8869a7bf9-cacerts\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:21:27.697813 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:27.697781 4560 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0486dfec-28d9-430e-8816-92f8869a7bf9-istio-csr-ca-configmap\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:21:27.697813 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:27.697794 4560 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0486dfec-28d9-430e-8816-92f8869a7bf9-istio-csr-dns-cert\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:21:27.697813 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:27.697807 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55stx\" (UniqueName: \"kubernetes.io/projected/0486dfec-28d9-430e-8816-92f8869a7bf9-kube-api-access-55stx\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:21:27.698061 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:27.697820 4560 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0486dfec-28d9-430e-8816-92f8869a7bf9-local-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:21:27.698061 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:27.697832 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0486dfec-28d9-430e-8816-92f8869a7bf9-istio-token\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:21:28.117141 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:28.117107 4560 generic.go:356] "Generic (PLEG): container finished" podID="0486dfec-28d9-430e-8816-92f8869a7bf9" containerID="5add80301a6ff4b6e35734b3162b0c948dc88c42349d519ce462ad865f3bcc04" exitCode=0 Oct 30 21:21:28.117617 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:28.117150 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn" Oct 30 21:21:28.117617 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:28.117193 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn" event={"ID":"0486dfec-28d9-430e-8816-92f8869a7bf9","Type":"ContainerDied","Data":"5add80301a6ff4b6e35734b3162b0c948dc88c42349d519ce462ad865f3bcc04"} Oct 30 21:21:28.117617 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:28.117250 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn" event={"ID":"0486dfec-28d9-430e-8816-92f8869a7bf9","Type":"ContainerDied","Data":"619df02ee468ce5bd45f6b15feebc79f1621ef1c06a1922e44e5fe90d04b505e"} Oct 30 21:21:28.117617 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:28.117278 4560 scope.go:122] "RemoveContainer" containerID="5add80301a6ff4b6e35734b3162b0c948dc88c42349d519ce462ad865f3bcc04" Oct 30 21:21:28.126033 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:28.126017 4560 scope.go:122] "RemoveContainer" containerID="5add80301a6ff4b6e35734b3162b0c948dc88c42349d519ce462ad865f3bcc04" Oct 30 21:21:28.126318 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:21:28.126299 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5add80301a6ff4b6e35734b3162b0c948dc88c42349d519ce462ad865f3bcc04\": container with ID starting with 5add80301a6ff4b6e35734b3162b0c948dc88c42349d519ce462ad865f3bcc04 not found: ID does not exist" containerID="5add80301a6ff4b6e35734b3162b0c948dc88c42349d519ce462ad865f3bcc04" Oct 30 21:21:28.126464 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:28.126326 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5add80301a6ff4b6e35734b3162b0c948dc88c42349d519ce462ad865f3bcc04"} err="failed to get container status \"5add80301a6ff4b6e35734b3162b0c948dc88c42349d519ce462ad865f3bcc04\": rpc error: code = NotFound desc = could not find container \"5add80301a6ff4b6e35734b3162b0c948dc88c42349d519ce462ad865f3bcc04\": container with ID starting with 5add80301a6ff4b6e35734b3162b0c948dc88c42349d519ce462ad865f3bcc04 not found: ID does not exist" Oct 30 21:21:28.271107 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:28.271077 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn"] Oct 30 21:21:28.317998 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:28.317957 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-bmwnn"] Oct 30 21:21:28.574303 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:28.574271 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0486dfec-28d9-430e-8816-92f8869a7bf9" path="/var/lib/kubelet/pods/0486dfec-28d9-430e-8816-92f8869a7bf9/volumes" Oct 30 21:21:29.641417 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:29.641389 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xkqx5"] Oct 30 21:21:29.641874 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:29.641769 4560 cpu_manager.go:414] "containerMap: removing container" podUID="0486dfec-28d9-430e-8816-92f8869a7bf9" containerName="discovery" Oct 30 21:21:29.641874 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:29.641785 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0486dfec-28d9-430e-8816-92f8869a7bf9" containerName="discovery" Oct 30 21:21:29.641874 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:29.641849 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="0486dfec-28d9-430e-8816-92f8869a7bf9" containerName="discovery" Oct 30 21:21:29.644750 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:29.644725 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xkqx5" Oct 30 21:21:29.695156 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:29.695129 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 21:21:29.695156 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:29.695142 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 21:21:29.699994 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:29.699971 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xkqx5"] Oct 30 21:21:29.708933 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:29.708911 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 21:21:29.708933 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:29.708912 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-rcm2v\"" Oct 30 21:21:29.717066 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:29.717042 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/afd6d37d-851e-4b57-9041-1409c864beae-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-xkqx5\" (UID: \"afd6d37d-851e-4b57-9041-1409c864beae\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xkqx5" Oct 30 21:21:29.717171 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:29.717077 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw57z\" (UniqueName: \"kubernetes.io/projected/afd6d37d-851e-4b57-9041-1409c864beae-kube-api-access-lw57z\") pod \"llmisvc-controller-manager-869d5dcc5d-xkqx5\" (UID: \"afd6d37d-851e-4b57-9041-1409c864beae\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xkqx5" Oct 30 21:21:29.750602 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:29.750577 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-ljvd8"] Oct 30 21:21:29.753517 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:29.753502 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-ljvd8" Oct 30 21:21:29.807847 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:29.807817 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-2vmvd\"" Oct 30 21:21:29.808076 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:29.807888 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 21:21:29.809030 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:29.809008 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-ljvd8"] Oct 30 21:21:29.818467 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:29.818447 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bae0774-2ade-4011-a36e-e0c36d5beee2-tls-certs\") pod \"model-serving-api-75bd89fdfc-ljvd8\" (UID: \"9bae0774-2ade-4011-a36e-e0c36d5beee2\") " pod="opendatahub/model-serving-api-75bd89fdfc-ljvd8" Oct 30 21:21:29.818566 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:29.818508 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/afd6d37d-851e-4b57-9041-1409c864beae-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-xkqx5\" (UID: \"afd6d37d-851e-4b57-9041-1409c864beae\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xkqx5" Oct 30 21:21:29.818616 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:29.818584 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw57z\" (UniqueName: \"kubernetes.io/projected/afd6d37d-851e-4b57-9041-1409c864beae-kube-api-access-lw57z\") pod \"llmisvc-controller-manager-869d5dcc5d-xkqx5\" (UID: \"afd6d37d-851e-4b57-9041-1409c864beae\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xkqx5" Oct 30 21:21:29.818695 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:29.818677 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rw4z\" (UniqueName: \"kubernetes.io/projected/9bae0774-2ade-4011-a36e-e0c36d5beee2-kube-api-access-4rw4z\") pod \"model-serving-api-75bd89fdfc-ljvd8\" (UID: \"9bae0774-2ade-4011-a36e-e0c36d5beee2\") " pod="opendatahub/model-serving-api-75bd89fdfc-ljvd8" Oct 30 21:21:29.820985 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:29.820963 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/afd6d37d-851e-4b57-9041-1409c864beae-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-xkqx5\" (UID: \"afd6d37d-851e-4b57-9041-1409c864beae\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xkqx5" Oct 30 21:21:29.876524 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:29.876495 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw57z\" (UniqueName: \"kubernetes.io/projected/afd6d37d-851e-4b57-9041-1409c864beae-kube-api-access-lw57z\") pod \"llmisvc-controller-manager-869d5dcc5d-xkqx5\" (UID: \"afd6d37d-851e-4b57-9041-1409c864beae\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xkqx5" Oct 30 21:21:29.919866 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:29.919798 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rw4z\" (UniqueName: \"kubernetes.io/projected/9bae0774-2ade-4011-a36e-e0c36d5beee2-kube-api-access-4rw4z\") pod \"model-serving-api-75bd89fdfc-ljvd8\" (UID: \"9bae0774-2ade-4011-a36e-e0c36d5beee2\") " pod="opendatahub/model-serving-api-75bd89fdfc-ljvd8" Oct 30 21:21:29.919866 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:29.919847 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bae0774-2ade-4011-a36e-e0c36d5beee2-tls-certs\") pod \"model-serving-api-75bd89fdfc-ljvd8\" (UID: \"9bae0774-2ade-4011-a36e-e0c36d5beee2\") " pod="opendatahub/model-serving-api-75bd89fdfc-ljvd8" Oct 30 21:21:29.922353 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:29.922311 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bae0774-2ade-4011-a36e-e0c36d5beee2-tls-certs\") pod \"model-serving-api-75bd89fdfc-ljvd8\" (UID: \"9bae0774-2ade-4011-a36e-e0c36d5beee2\") " pod="opendatahub/model-serving-api-75bd89fdfc-ljvd8" Oct 30 21:21:29.956074 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:29.956049 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xkqx5" Oct 30 21:21:29.978904 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:29.978877 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rw4z\" (UniqueName: \"kubernetes.io/projected/9bae0774-2ade-4011-a36e-e0c36d5beee2-kube-api-access-4rw4z\") pod \"model-serving-api-75bd89fdfc-ljvd8\" (UID: \"9bae0774-2ade-4011-a36e-e0c36d5beee2\") " pod="opendatahub/model-serving-api-75bd89fdfc-ljvd8" Oct 30 21:21:30.066266 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:30.066234 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-ljvd8" Oct 30 21:21:30.332381 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:30.332228 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xkqx5"] Oct 30 21:21:30.338327 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:21:30.338304 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podafd6d37d_851e_4b57_9041_1409c864beae.slice/crio-e6c55507e2667c86a1b78b6032c881611b8473e331c40028256fda557b820367 WatchSource:0}: Error finding container e6c55507e2667c86a1b78b6032c881611b8473e331c40028256fda557b820367: Status 404 returned error can't find the container with id e6c55507e2667c86a1b78b6032c881611b8473e331c40028256fda557b820367 Oct 30 21:21:30.432999 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:30.432971 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-ljvd8"] Oct 30 21:21:30.439444 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:21:30.439417 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bae0774_2ade_4011_a36e_e0c36d5beee2.slice/crio-ed4e92e69ed135ae5c12acaae46202f3fbf03f30e271d037505b262b39d8b624 WatchSource:0}: Error finding container ed4e92e69ed135ae5c12acaae46202f3fbf03f30e271d037505b262b39d8b624: Status 404 returned error can't find the container with id ed4e92e69ed135ae5c12acaae46202f3fbf03f30e271d037505b262b39d8b624 Oct 30 21:21:31.131283 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:31.131243 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xkqx5" event={"ID":"afd6d37d-851e-4b57-9041-1409c864beae","Type":"ContainerStarted","Data":"e6c55507e2667c86a1b78b6032c881611b8473e331c40028256fda557b820367"} Oct 30 21:21:31.132251 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:31.132227 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-ljvd8" event={"ID":"9bae0774-2ade-4011-a36e-e0c36d5beee2","Type":"ContainerStarted","Data":"ed4e92e69ed135ae5c12acaae46202f3fbf03f30e271d037505b262b39d8b624"} Oct 30 21:21:33.144449 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:33.144406 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-ljvd8" event={"ID":"9bae0774-2ade-4011-a36e-e0c36d5beee2","Type":"ContainerStarted","Data":"4d1b28107f320b3d4d4b8bfca35e613e8d82e22cc9d61d4e2ee49b489713ab3d"} Oct 30 21:21:33.144931 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:33.144575 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-ljvd8" Oct 30 21:21:33.255446 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:33.255400 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-ljvd8" podStartSLOduration=1.7688830979999999 podStartE2EDuration="4.255386414s" totalImagesPullingTime="2.486503316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:33.254559096 +0000 UTC m=+468.809258874" watchObservedRunningTime="2026-10-30 21:21:33.255386414 +0000 UTC m=+468.810086191" Oct 30 21:21:34.150179 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:34.150142 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xkqx5" event={"ID":"afd6d37d-851e-4b57-9041-1409c864beae","Type":"ContainerStarted","Data":"7cd4b0b2f6294f76ef36684e5def2e9b59bba18a11644f9088c1ed6e4eb6e29c"} Oct 30 21:21:34.150609 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:34.150284 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xkqx5" Oct 30 21:21:34.262847 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:34.262798 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xkqx5" podStartSLOduration=2.324533329 podStartE2EDuration="5.262784005s" totalImagesPullingTime="2.938250676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:34.260968962 +0000 UTC m=+469.815668740" watchObservedRunningTime="2026-10-30 21:21:34.262784005 +0000 UTC m=+469.817483831" Oct 30 21:21:44.154360 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:21:44.154306 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-ljvd8" Oct 30 21:21:49.683944 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:21:49.683903 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:22:05.156015 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:22:05.155943 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xkqx5" Oct 30 21:22:19.683263 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:22:19.683226 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:22:49.682818 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:22:49.682781 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:23:19.684490 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:23:19.684380 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:23:49.683515 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:23:49.683437 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:24:19.683901 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:24:19.683853 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:24:49.683767 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:24:49.683731 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:25:19.683963 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:25:19.683880 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:25:49.683935 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:25:49.683895 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:26:19.683685 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:26:19.683644 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:26:49.683894 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:26:49.683810 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:27:19.682474 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:27:19.682437 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:27:49.682791 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:27:49.682757 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:28:19.684168 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:28:19.684133 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:28:49.682244 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:28:49.682211 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:29:19.683258 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:29:19.683219 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:29:49.682735 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:29:49.682657 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:30:19.683648 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:30:19.683607 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:30:49.684396 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:30:49.684357 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:30:51.680530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:51.680498 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-ft7rc"] Oct 30 21:30:51.683480 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:51.683452 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ft7rc" Oct 30 21:30:51.737601 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:51.737569 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 30 21:30:51.737743 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:51.737679 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 21:30:51.784933 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:51.784898 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7dnw\" (UniqueName: \"kubernetes.io/projected/ba6cc181-9632-4861-ab91-0e0836e16f49-kube-api-access-x7dnw\") pod \"kuadrant-console-plugin-66c4fb575d-ft7rc\" (UID: \"ba6cc181-9632-4861-ab91-0e0836e16f49\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ft7rc" Oct 30 21:30:51.785054 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:51.784938 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ba6cc181-9632-4861-ab91-0e0836e16f49-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-ft7rc\" (UID: \"ba6cc181-9632-4861-ab91-0e0836e16f49\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ft7rc" Oct 30 21:30:51.785097 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:51.785042 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ba6cc181-9632-4861-ab91-0e0836e16f49-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-ft7rc\" (UID: \"ba6cc181-9632-4861-ab91-0e0836e16f49\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ft7rc" Oct 30 21:30:51.788028 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:51.788002 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 21:30:51.788207 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:51.788187 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-b945v\"" Oct 30 21:30:51.788270 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:51.788191 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 30 21:30:51.789203 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:51.789186 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-ft7rc"] Oct 30 21:30:51.886161 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:51.886133 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ba6cc181-9632-4861-ab91-0e0836e16f49-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-ft7rc\" (UID: \"ba6cc181-9632-4861-ab91-0e0836e16f49\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ft7rc" Oct 30 21:30:51.886294 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:51.886189 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7dnw\" (UniqueName: \"kubernetes.io/projected/ba6cc181-9632-4861-ab91-0e0836e16f49-kube-api-access-x7dnw\") pod \"kuadrant-console-plugin-66c4fb575d-ft7rc\" (UID: \"ba6cc181-9632-4861-ab91-0e0836e16f49\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ft7rc" Oct 30 21:30:51.886294 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:51.886211 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ba6cc181-9632-4861-ab91-0e0836e16f49-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-ft7rc\" (UID: \"ba6cc181-9632-4861-ab91-0e0836e16f49\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ft7rc" Oct 30 21:30:51.886934 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:51.886904 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ba6cc181-9632-4861-ab91-0e0836e16f49-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-ft7rc\" (UID: \"ba6cc181-9632-4861-ab91-0e0836e16f49\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ft7rc" Oct 30 21:30:51.888667 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:51.888645 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ba6cc181-9632-4861-ab91-0e0836e16f49-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-ft7rc\" (UID: \"ba6cc181-9632-4861-ab91-0e0836e16f49\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ft7rc" Oct 30 21:30:51.948960 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:51.948894 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7dnw\" (UniqueName: \"kubernetes.io/projected/ba6cc181-9632-4861-ab91-0e0836e16f49-kube-api-access-x7dnw\") pod \"kuadrant-console-plugin-66c4fb575d-ft7rc\" (UID: \"ba6cc181-9632-4861-ab91-0e0836e16f49\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ft7rc" Oct 30 21:30:52.007448 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:52.007423 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ft7rc" Oct 30 21:30:52.381002 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:52.380979 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-ft7rc"] Oct 30 21:30:52.383882 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:30:52.383853 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba6cc181_9632_4861_ab91_0e0836e16f49.slice/crio-3afbd519f0ceaa1cca19c172c1af1b51b7b6067e646253912ae7f14e2ac66fa0 WatchSource:0}: Error finding container 3afbd519f0ceaa1cca19c172c1af1b51b7b6067e646253912ae7f14e2ac66fa0: Status 404 returned error can't find the container with id 3afbd519f0ceaa1cca19c172c1af1b51b7b6067e646253912ae7f14e2ac66fa0 Oct 30 21:30:52.385616 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:52.385600 4560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:30:53.003497 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:53.003466 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ft7rc" event={"ID":"ba6cc181-9632-4861-ab91-0e0836e16f49","Type":"ContainerStarted","Data":"3afbd519f0ceaa1cca19c172c1af1b51b7b6067e646253912ae7f14e2ac66fa0"} Oct 30 21:30:53.240205 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:53.240167 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pcs8w"] Oct 30 21:30:53.244448 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:53.244432 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-pcs8w" Oct 30 21:30:53.295401 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:53.295332 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pcs8w"] Oct 30 21:30:53.344728 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:53.344696 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-klfvt\"" Oct 30 21:30:53.398483 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:53.398453 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqswd\" (UniqueName: \"kubernetes.io/projected/b394bdd3-951b-44df-a69b-efbe7e3dfc7a-kube-api-access-cqswd\") pod \"authorino-57f7f8659f-pcs8w\" (UID: \"b394bdd3-951b-44df-a69b-efbe7e3dfc7a\") " pod="kuadrant-system/authorino-57f7f8659f-pcs8w" Oct 30 21:30:53.499319 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:53.499283 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqswd\" (UniqueName: \"kubernetes.io/projected/b394bdd3-951b-44df-a69b-efbe7e3dfc7a-kube-api-access-cqswd\") pod \"authorino-57f7f8659f-pcs8w\" (UID: \"b394bdd3-951b-44df-a69b-efbe7e3dfc7a\") " pod="kuadrant-system/authorino-57f7f8659f-pcs8w" Oct 30 21:30:53.561156 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:53.561092 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqswd\" (UniqueName: \"kubernetes.io/projected/b394bdd3-951b-44df-a69b-efbe7e3dfc7a-kube-api-access-cqswd\") pod \"authorino-57f7f8659f-pcs8w\" (UID: \"b394bdd3-951b-44df-a69b-efbe7e3dfc7a\") " pod="kuadrant-system/authorino-57f7f8659f-pcs8w" Oct 30 21:30:53.856117 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:53.856044 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-pcs8w" Oct 30 21:30:54.224395 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:54.224318 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pcs8w"] Oct 30 21:30:54.227716 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:30:54.227685 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb394bdd3_951b_44df_a69b_efbe7e3dfc7a.slice/crio-29a3f89eae2f0512f00b0622866cec4b4a013f1cb955b4a157fd76abb82f0a8d WatchSource:0}: Error finding container 29a3f89eae2f0512f00b0622866cec4b4a013f1cb955b4a157fd76abb82f0a8d: Status 404 returned error can't find the container with id 29a3f89eae2f0512f00b0622866cec4b4a013f1cb955b4a157fd76abb82f0a8d Oct 30 21:30:54.984534 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:54.984495 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:30:54.987852 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:54.987822 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-475wg" Oct 30 21:30:55.013324 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:55.013296 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-pcs8w" event={"ID":"b394bdd3-951b-44df-a69b-efbe7e3dfc7a","Type":"ContainerStarted","Data":"29a3f89eae2f0512f00b0622866cec4b4a013f1cb955b4a157fd76abb82f0a8d"} Oct 30 21:30:55.094249 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:55.094214 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:30:55.113707 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:55.113677 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8nrr\" (UniqueName: \"kubernetes.io/projected/b4ea7858-6e8b-4fa8-a805-0d82678b5dac-kube-api-access-t8nrr\") pod \"limitador-limitador-7c9b87d574-475wg\" (UID: \"b4ea7858-6e8b-4fa8-a805-0d82678b5dac\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-475wg" Oct 30 21:30:55.113832 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:55.113717 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b4ea7858-6e8b-4fa8-a805-0d82678b5dac-config-file\") pod \"limitador-limitador-7c9b87d574-475wg\" (UID: \"b4ea7858-6e8b-4fa8-a805-0d82678b5dac\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-475wg" Oct 30 21:30:55.139942 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:55.139915 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 21:30:55.214593 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:55.214555 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8nrr\" (UniqueName: \"kubernetes.io/projected/b4ea7858-6e8b-4fa8-a805-0d82678b5dac-kube-api-access-t8nrr\") pod \"limitador-limitador-7c9b87d574-475wg\" (UID: \"b4ea7858-6e8b-4fa8-a805-0d82678b5dac\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-475wg" Oct 30 21:30:55.214798 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:55.214612 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b4ea7858-6e8b-4fa8-a805-0d82678b5dac-config-file\") pod \"limitador-limitador-7c9b87d574-475wg\" (UID: \"b4ea7858-6e8b-4fa8-a805-0d82678b5dac\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-475wg" Oct 30 21:30:55.215361 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:55.215322 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b4ea7858-6e8b-4fa8-a805-0d82678b5dac-config-file\") pod \"limitador-limitador-7c9b87d574-475wg\" (UID: \"b4ea7858-6e8b-4fa8-a805-0d82678b5dac\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-475wg" Oct 30 21:30:55.264899 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:55.264825 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:30:55.276769 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:55.276740 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8nrr\" (UniqueName: \"kubernetes.io/projected/b4ea7858-6e8b-4fa8-a805-0d82678b5dac-kube-api-access-t8nrr\") pod \"limitador-limitador-7c9b87d574-475wg\" (UID: \"b4ea7858-6e8b-4fa8-a805-0d82678b5dac\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-475wg" Oct 30 21:30:55.303506 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:55.303481 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-475wg" Oct 30 21:30:55.687596 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:55.687526 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:30:55.693474 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:30:55.693446 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4ea7858_6e8b_4fa8_a805_0d82678b5dac.slice/crio-3dab197ade1c8634269e035219ef9f7c9565a551240ca6bd3627898cfefa6903 WatchSource:0}: Error finding container 3dab197ade1c8634269e035219ef9f7c9565a551240ca6bd3627898cfefa6903: Status 404 returned error can't find the container with id 3dab197ade1c8634269e035219ef9f7c9565a551240ca6bd3627898cfefa6903 Oct 30 21:30:56.020198 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:56.020159 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-475wg" event={"ID":"b4ea7858-6e8b-4fa8-a805-0d82678b5dac","Type":"ContainerStarted","Data":"3dab197ade1c8634269e035219ef9f7c9565a551240ca6bd3627898cfefa6903"} Oct 30 21:30:59.034061 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:59.034028 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ft7rc" event={"ID":"ba6cc181-9632-4861-ab91-0e0836e16f49","Type":"ContainerStarted","Data":"1d4424a80e3354e40c72ffa9247d420692d90ddb25754fd9ca2030617db5aa5d"} Oct 30 21:30:59.035582 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:59.035556 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-475wg" event={"ID":"b4ea7858-6e8b-4fa8-a805-0d82678b5dac","Type":"ContainerStarted","Data":"00cc21dc37d86c61cae8504c526f2b1836c52e27f11381e274793363ec9f3c7b"} Oct 30 21:30:59.035683 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:59.035659 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-475wg" Oct 30 21:30:59.036984 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:59.036961 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-pcs8w" event={"ID":"b394bdd3-951b-44df-a69b-efbe7e3dfc7a","Type":"ContainerStarted","Data":"983b00f06028b590dd861b7308c585822ba14d3c6828fa28993dec37ac48e5e0"} Oct 30 21:30:59.150901 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:59.150807 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ft7rc" podStartSLOduration=1.721062012 podStartE2EDuration="8.150795221s" totalImagesPullingTime="6.429733209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:59.150478037 +0000 UTC m=+1034.705177814" watchObservedRunningTime="2026-10-30 21:30:59.150795221 +0000 UTC m=+1034.705494997" Oct 30 21:30:59.269719 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:59.269658 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-475wg" podStartSLOduration=2.03642826 podStartE2EDuration="5.269638993s" totalImagesPullingTime="3.233210733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:59.26902796 +0000 UTC m=+1034.823727737" watchObservedRunningTime="2026-10-30 21:30:59.269638993 +0000 UTC m=+1034.824338772" Oct 30 21:30:59.385258 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:30:59.385206 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-pcs8w" podStartSLOduration=2.250444816 podStartE2EDuration="6.38519117s" totalImagesPullingTime="4.134746354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:59.384245806 +0000 UTC m=+1034.938945583" watchObservedRunningTime="2026-10-30 21:30:59.38519117 +0000 UTC m=+1034.939890964" Oct 30 21:31:00.206995 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:31:00.206965 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pcs8w"] Oct 30 21:31:01.044888 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:31:01.044830 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-pcs8w" podUID="b394bdd3-951b-44df-a69b-efbe7e3dfc7a" containerName="authorino" containerID="cri-o://983b00f06028b590dd861b7308c585822ba14d3c6828fa28993dec37ac48e5e0" gracePeriod=30 Oct 30 21:31:01.277619 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:31:01.277594 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-pcs8w" Oct 30 21:31:01.467571 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:31:01.467504 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b394bdd3-951b-44df-a69b-efbe7e3dfc7a-kube-api-access-cqswd\" (UniqueName: \"kubernetes.io/projected/b394bdd3-951b-44df-a69b-efbe7e3dfc7a-kube-api-access-cqswd\") pod \"b394bdd3-951b-44df-a69b-efbe7e3dfc7a\" (UID: \"b394bdd3-951b-44df-a69b-efbe7e3dfc7a\") " Oct 30 21:31:01.469688 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:31:01.469666 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b394bdd3-951b-44df-a69b-efbe7e3dfc7a-kube-api-access-cqswd" pod "b394bdd3-951b-44df-a69b-efbe7e3dfc7a" (UID: "b394bdd3-951b-44df-a69b-efbe7e3dfc7a"). InnerVolumeSpecName "kube-api-access-cqswd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:01.568174 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:31:01.568144 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqswd\" (UniqueName: \"kubernetes.io/projected/b394bdd3-951b-44df-a69b-efbe7e3dfc7a-kube-api-access-cqswd\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:02.050302 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:31:02.050267 4560 generic.go:356] "Generic (PLEG): container finished" podID="b394bdd3-951b-44df-a69b-efbe7e3dfc7a" containerID="983b00f06028b590dd861b7308c585822ba14d3c6828fa28993dec37ac48e5e0" exitCode=0 Oct 30 21:31:02.050487 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:31:02.050309 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-pcs8w" event={"ID":"b394bdd3-951b-44df-a69b-efbe7e3dfc7a","Type":"ContainerDied","Data":"983b00f06028b590dd861b7308c585822ba14d3c6828fa28993dec37ac48e5e0"} Oct 30 21:31:02.050487 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:31:02.050318 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-pcs8w" Oct 30 21:31:02.050487 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:31:02.050370 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-pcs8w" event={"ID":"b394bdd3-951b-44df-a69b-efbe7e3dfc7a","Type":"ContainerDied","Data":"29a3f89eae2f0512f00b0622866cec4b4a013f1cb955b4a157fd76abb82f0a8d"} Oct 30 21:31:02.050487 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:31:02.050393 4560 scope.go:122] "RemoveContainer" containerID="983b00f06028b590dd861b7308c585822ba14d3c6828fa28993dec37ac48e5e0" Oct 30 21:31:02.058751 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:31:02.058730 4560 scope.go:122] "RemoveContainer" containerID="983b00f06028b590dd861b7308c585822ba14d3c6828fa28993dec37ac48e5e0" Oct 30 21:31:02.059046 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:31:02.059019 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"983b00f06028b590dd861b7308c585822ba14d3c6828fa28993dec37ac48e5e0\": container with ID starting with 983b00f06028b590dd861b7308c585822ba14d3c6828fa28993dec37ac48e5e0 not found: ID does not exist" containerID="983b00f06028b590dd861b7308c585822ba14d3c6828fa28993dec37ac48e5e0" Oct 30 21:31:02.059095 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:31:02.059056 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"983b00f06028b590dd861b7308c585822ba14d3c6828fa28993dec37ac48e5e0"} err="failed to get container status \"983b00f06028b590dd861b7308c585822ba14d3c6828fa28993dec37ac48e5e0\": rpc error: code = NotFound desc = could not find container \"983b00f06028b590dd861b7308c585822ba14d3c6828fa28993dec37ac48e5e0\": container with ID starting with 983b00f06028b590dd861b7308c585822ba14d3c6828fa28993dec37ac48e5e0 not found: ID does not exist" Oct 30 21:31:02.215750 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:31:02.215721 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pcs8w"] Oct 30 21:31:02.219295 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:31:02.219274 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pcs8w"] Oct 30 21:31:02.573645 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:31:02.573613 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b394bdd3-951b-44df-a69b-efbe7e3dfc7a" path="/var/lib/kubelet/pods/b394bdd3-951b-44df-a69b-efbe7e3dfc7a/volumes" Oct 30 21:31:10.041181 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:31:10.041150 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-475wg" Oct 30 21:31:19.682641 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:31:19.682599 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:31:49.682859 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:31:49.682812 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:32:19.683113 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:32:19.683067 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:32:26.227553 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:26.227510 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:32:31.645606 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:31.645529 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-cbc9ccf97-gqg6m"] Oct 30 21:32:31.645969 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:31.645801 4560 cpu_manager.go:414] "containerMap: removing container" podUID="b394bdd3-951b-44df-a69b-efbe7e3dfc7a" containerName="authorino" Oct 30 21:32:31.645969 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:31.645812 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b394bdd3-951b-44df-a69b-efbe7e3dfc7a" containerName="authorino" Oct 30 21:32:31.645969 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:31.645877 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="b394bdd3-951b-44df-a69b-efbe7e3dfc7a" containerName="authorino" Oct 30 21:32:31.648724 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:31.648708 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cbc9ccf97-gqg6m" Oct 30 21:32:31.707014 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:31.706978 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-cbc9ccf97-gqg6m"] Oct 30 21:32:31.707184 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:31.707113 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-klfvt\"" Oct 30 21:32:31.729587 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:31.729563 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r4wv\" (UniqueName: \"kubernetes.io/projected/e4e5b008-1a97-4e70-a9bd-ac4389a1123e-kube-api-access-9r4wv\") pod \"authorino-cbc9ccf97-gqg6m\" (UID: \"e4e5b008-1a97-4e70-a9bd-ac4389a1123e\") " pod="kuadrant-system/authorino-cbc9ccf97-gqg6m" Oct 30 21:32:31.830228 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:31.830198 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r4wv\" (UniqueName: \"kubernetes.io/projected/e4e5b008-1a97-4e70-a9bd-ac4389a1123e-kube-api-access-9r4wv\") pod \"authorino-cbc9ccf97-gqg6m\" (UID: \"e4e5b008-1a97-4e70-a9bd-ac4389a1123e\") " pod="kuadrant-system/authorino-cbc9ccf97-gqg6m" Oct 30 21:32:31.893534 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:31.893503 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r4wv\" (UniqueName: \"kubernetes.io/projected/e4e5b008-1a97-4e70-a9bd-ac4389a1123e-kube-api-access-9r4wv\") pod \"authorino-cbc9ccf97-gqg6m\" (UID: \"e4e5b008-1a97-4e70-a9bd-ac4389a1123e\") " pod="kuadrant-system/authorino-cbc9ccf97-gqg6m" Oct 30 21:32:31.959487 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:31.959419 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cbc9ccf97-gqg6m" Oct 30 21:32:32.040234 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:32.040209 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-cbc9ccf97-gqg6m"] Oct 30 21:32:32.330272 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:32.330244 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-cbc9ccf97-gqg6m"] Oct 30 21:32:32.333983 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:32:32.333957 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4e5b008_1a97_4e70_a9bd_ac4389a1123e.slice/crio-294bbdda83386a3a1853067fa6f17678ab405fb23e970bd53f830aa8865a73cf WatchSource:0}: Error finding container 294bbdda83386a3a1853067fa6f17678ab405fb23e970bd53f830aa8865a73cf: Status 404 returned error can't find the container with id 294bbdda83386a3a1853067fa6f17678ab405fb23e970bd53f830aa8865a73cf Oct 30 21:32:32.379392 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:32.379359 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cbc9ccf97-gqg6m" event={"ID":"e4e5b008-1a97-4e70-a9bd-ac4389a1123e","Type":"ContainerStarted","Data":"294bbdda83386a3a1853067fa6f17678ab405fb23e970bd53f830aa8865a73cf"} Oct 30 21:32:34.388455 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:34.388416 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cbc9ccf97-gqg6m" event={"ID":"e4e5b008-1a97-4e70-a9bd-ac4389a1123e","Type":"ContainerStarted","Data":"d9436560af2ff935119395824d684f395a1887b52585be67c8fd829071d6348a"} Oct 30 21:32:34.388878 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:34.388452 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-cbc9ccf97-gqg6m" podUID="e4e5b008-1a97-4e70-a9bd-ac4389a1123e" containerName="authorino" containerID="cri-o://d9436560af2ff935119395824d684f395a1887b52585be67c8fd829071d6348a" gracePeriod=30 Oct 30 21:32:34.513465 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:34.513419 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-cbc9ccf97-gqg6m" podStartSLOduration=1.936850485 podStartE2EDuration="3.513404406s" totalImagesPullingTime="1.576553921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:34.512805013 +0000 UTC m=+1130.067504790" watchObservedRunningTime="2026-10-30 21:32:34.513404406 +0000 UTC m=+1130.068104183" Oct 30 21:32:34.638639 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:34.638586 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cbc9ccf97-gqg6m" Oct 30 21:32:34.649764 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:34.649741 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4e5b008-1a97-4e70-a9bd-ac4389a1123e-kube-api-access-9r4wv\" (UniqueName: \"kubernetes.io/projected/e4e5b008-1a97-4e70-a9bd-ac4389a1123e-kube-api-access-9r4wv\") pod \"e4e5b008-1a97-4e70-a9bd-ac4389a1123e\" (UID: \"e4e5b008-1a97-4e70-a9bd-ac4389a1123e\") " Oct 30 21:32:34.651619 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:34.651597 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4e5b008-1a97-4e70-a9bd-ac4389a1123e-kube-api-access-9r4wv" pod "e4e5b008-1a97-4e70-a9bd-ac4389a1123e" (UID: "e4e5b008-1a97-4e70-a9bd-ac4389a1123e"). InnerVolumeSpecName "kube-api-access-9r4wv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:34.750302 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:34.750270 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r4wv\" (UniqueName: \"kubernetes.io/projected/e4e5b008-1a97-4e70-a9bd-ac4389a1123e-kube-api-access-9r4wv\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:35.392613 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:35.392571 4560 generic.go:356] "Generic (PLEG): container finished" podID="e4e5b008-1a97-4e70-a9bd-ac4389a1123e" containerID="d9436560af2ff935119395824d684f395a1887b52585be67c8fd829071d6348a" exitCode=0 Oct 30 21:32:35.393056 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:35.392620 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cbc9ccf97-gqg6m" Oct 30 21:32:35.393056 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:35.392617 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cbc9ccf97-gqg6m" event={"ID":"e4e5b008-1a97-4e70-a9bd-ac4389a1123e","Type":"ContainerDied","Data":"d9436560af2ff935119395824d684f395a1887b52585be67c8fd829071d6348a"} Oct 30 21:32:35.393056 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:35.392663 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cbc9ccf97-gqg6m" event={"ID":"e4e5b008-1a97-4e70-a9bd-ac4389a1123e","Type":"ContainerDied","Data":"294bbdda83386a3a1853067fa6f17678ab405fb23e970bd53f830aa8865a73cf"} Oct 30 21:32:35.393056 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:35.392682 4560 scope.go:122] "RemoveContainer" containerID="d9436560af2ff935119395824d684f395a1887b52585be67c8fd829071d6348a" Oct 30 21:32:35.402840 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:35.402819 4560 scope.go:122] "RemoveContainer" containerID="d9436560af2ff935119395824d684f395a1887b52585be67c8fd829071d6348a" Oct 30 21:32:35.403086 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:32:35.403067 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9436560af2ff935119395824d684f395a1887b52585be67c8fd829071d6348a\": container with ID starting with d9436560af2ff935119395824d684f395a1887b52585be67c8fd829071d6348a not found: ID does not exist" containerID="d9436560af2ff935119395824d684f395a1887b52585be67c8fd829071d6348a" Oct 30 21:32:35.403126 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:35.403094 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9436560af2ff935119395824d684f395a1887b52585be67c8fd829071d6348a"} err="failed to get container status \"d9436560af2ff935119395824d684f395a1887b52585be67c8fd829071d6348a\": rpc error: code = NotFound desc = could not find container \"d9436560af2ff935119395824d684f395a1887b52585be67c8fd829071d6348a\": container with ID starting with d9436560af2ff935119395824d684f395a1887b52585be67c8fd829071d6348a not found: ID does not exist" Oct 30 21:32:35.573711 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:35.573682 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-cbc9ccf97-gqg6m"] Oct 30 21:32:35.623889 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:35.623861 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-cbc9ccf97-gqg6m"] Oct 30 21:32:35.752033 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:35.751997 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-5gk2d"] Oct 30 21:32:35.752316 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:35.752303 4560 cpu_manager.go:414] "containerMap: removing container" podUID="e4e5b008-1a97-4e70-a9bd-ac4389a1123e" containerName="authorino" Oct 30 21:32:35.752384 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:35.752317 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4e5b008-1a97-4e70-a9bd-ac4389a1123e" containerName="authorino" Oct 30 21:32:35.752421 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:35.752390 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4e5b008-1a97-4e70-a9bd-ac4389a1123e" containerName="authorino" Oct 30 21:32:35.756278 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:35.756262 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5gk2d" Oct 30 21:32:35.813025 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:35.812992 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-5gk2d"] Oct 30 21:32:35.858173 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:35.858142 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12fcc260-c72a-44da-a01a-f654ccbe1c6e-cert\") pod \"llmisvc-controller-manager-6466c554d8-5gk2d\" (UID: \"12fcc260-c72a-44da-a01a-f654ccbe1c6e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5gk2d" Oct 30 21:32:35.858286 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:35.858198 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qm6n\" (UniqueName: \"kubernetes.io/projected/12fcc260-c72a-44da-a01a-f654ccbe1c6e-kube-api-access-4qm6n\") pod \"llmisvc-controller-manager-6466c554d8-5gk2d\" (UID: \"12fcc260-c72a-44da-a01a-f654ccbe1c6e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5gk2d" Oct 30 21:32:35.958598 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:35.958564 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12fcc260-c72a-44da-a01a-f654ccbe1c6e-cert\") pod \"llmisvc-controller-manager-6466c554d8-5gk2d\" (UID: \"12fcc260-c72a-44da-a01a-f654ccbe1c6e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5gk2d" Oct 30 21:32:35.958756 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:35.958639 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qm6n\" (UniqueName: \"kubernetes.io/projected/12fcc260-c72a-44da-a01a-f654ccbe1c6e-kube-api-access-4qm6n\") pod \"llmisvc-controller-manager-6466c554d8-5gk2d\" (UID: \"12fcc260-c72a-44da-a01a-f654ccbe1c6e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5gk2d" Oct 30 21:32:35.961085 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:35.961054 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12fcc260-c72a-44da-a01a-f654ccbe1c6e-cert\") pod \"llmisvc-controller-manager-6466c554d8-5gk2d\" (UID: \"12fcc260-c72a-44da-a01a-f654ccbe1c6e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5gk2d" Oct 30 21:32:36.023832 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:36.023769 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qm6n\" (UniqueName: \"kubernetes.io/projected/12fcc260-c72a-44da-a01a-f654ccbe1c6e-kube-api-access-4qm6n\") pod \"llmisvc-controller-manager-6466c554d8-5gk2d\" (UID: \"12fcc260-c72a-44da-a01a-f654ccbe1c6e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5gk2d" Oct 30 21:32:36.068559 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:36.068534 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5gk2d" Oct 30 21:32:36.439807 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:36.439740 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-5gk2d"] Oct 30 21:32:36.442875 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:32:36.442849 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12fcc260_c72a_44da_a01a_f654ccbe1c6e.slice/crio-25fccf1ff38def9cad6f78fc7d4af9303088c6c4ee83fbbf4f799f4ae5e07fa0 WatchSource:0}: Error finding container 25fccf1ff38def9cad6f78fc7d4af9303088c6c4ee83fbbf4f799f4ae5e07fa0: Status 404 returned error can't find the container with id 25fccf1ff38def9cad6f78fc7d4af9303088c6c4ee83fbbf4f799f4ae5e07fa0 Oct 30 21:32:36.573974 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:36.573949 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4e5b008-1a97-4e70-a9bd-ac4389a1123e" path="/var/lib/kubelet/pods/e4e5b008-1a97-4e70-a9bd-ac4389a1123e/volumes" Oct 30 21:32:37.402998 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:37.402961 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5gk2d" event={"ID":"12fcc260-c72a-44da-a01a-f654ccbe1c6e","Type":"ContainerStarted","Data":"3007a4e082dfb4fa42f2de1ef91df318b2f73b5a2531f0abc9f622467aeb5bb9"} Oct 30 21:32:37.403169 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:37.403019 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5gk2d" event={"ID":"12fcc260-c72a-44da-a01a-f654ccbe1c6e","Type":"ContainerStarted","Data":"25fccf1ff38def9cad6f78fc7d4af9303088c6c4ee83fbbf4f799f4ae5e07fa0"} Oct 30 21:32:37.403169 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:37.403065 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5gk2d" Oct 30 21:32:37.522897 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:32:37.522837 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5gk2d" podStartSLOduration=1.750131184 podStartE2EDuration="2.522818375s" totalImagesPullingTime="772.687191ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:37.52206842 +0000 UTC m=+1133.076768198" watchObservedRunningTime="2026-10-30 21:32:37.522818375 +0000 UTC m=+1133.077518189" Oct 30 21:32:49.684057 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:32:49.684009 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:33:08.409285 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:08.409249 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5gk2d" Oct 30 21:33:08.580897 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:08.580856 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xkqx5"] Oct 30 21:33:08.581144 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:08.581105 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xkqx5" podUID="afd6d37d-851e-4b57-9041-1409c864beae" containerName="manager" containerID="cri-o://7cd4b0b2f6294f76ef36684e5def2e9b59bba18a11644f9088c1ed6e4eb6e29c" gracePeriod=30 Oct 30 21:33:08.818024 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:08.817999 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xkqx5" Oct 30 21:33:08.920944 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:08.920911 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afd6d37d-851e-4b57-9041-1409c864beae-cert\" (UniqueName: \"kubernetes.io/secret/afd6d37d-851e-4b57-9041-1409c864beae-cert\") pod \"afd6d37d-851e-4b57-9041-1409c864beae\" (UID: \"afd6d37d-851e-4b57-9041-1409c864beae\") " Oct 30 21:33:08.921079 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:08.920980 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afd6d37d-851e-4b57-9041-1409c864beae-kube-api-access-lw57z\" (UniqueName: \"kubernetes.io/projected/afd6d37d-851e-4b57-9041-1409c864beae-kube-api-access-lw57z\") pod \"afd6d37d-851e-4b57-9041-1409c864beae\" (UID: \"afd6d37d-851e-4b57-9041-1409c864beae\") " Oct 30 21:33:08.922897 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:08.922873 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd6d37d-851e-4b57-9041-1409c864beae-kube-api-access-lw57z" pod "afd6d37d-851e-4b57-9041-1409c864beae" (UID: "afd6d37d-851e-4b57-9041-1409c864beae"). InnerVolumeSpecName "kube-api-access-lw57z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:33:08.922998 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:08.922932 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afd6d37d-851e-4b57-9041-1409c864beae-cert" pod "afd6d37d-851e-4b57-9041-1409c864beae" (UID: "afd6d37d-851e-4b57-9041-1409c864beae"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:33:09.021738 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:09.021701 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw57z\" (UniqueName: \"kubernetes.io/projected/afd6d37d-851e-4b57-9041-1409c864beae-kube-api-access-lw57z\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:09.021738 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:09.021733 4560 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/afd6d37d-851e-4b57-9041-1409c864beae-cert\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:09.521776 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:09.521745 4560 generic.go:356] "Generic (PLEG): container finished" podID="afd6d37d-851e-4b57-9041-1409c864beae" containerID="7cd4b0b2f6294f76ef36684e5def2e9b59bba18a11644f9088c1ed6e4eb6e29c" exitCode=0 Oct 30 21:33:09.522206 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:09.521791 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xkqx5" Oct 30 21:33:09.522206 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:09.521831 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xkqx5" event={"ID":"afd6d37d-851e-4b57-9041-1409c864beae","Type":"ContainerDied","Data":"7cd4b0b2f6294f76ef36684e5def2e9b59bba18a11644f9088c1ed6e4eb6e29c"} Oct 30 21:33:09.522206 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:09.521881 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xkqx5" event={"ID":"afd6d37d-851e-4b57-9041-1409c864beae","Type":"ContainerDied","Data":"e6c55507e2667c86a1b78b6032c881611b8473e331c40028256fda557b820367"} Oct 30 21:33:09.522206 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:09.521903 4560 scope.go:122] "RemoveContainer" containerID="7cd4b0b2f6294f76ef36684e5def2e9b59bba18a11644f9088c1ed6e4eb6e29c" Oct 30 21:33:09.530381 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:09.530366 4560 scope.go:122] "RemoveContainer" containerID="7cd4b0b2f6294f76ef36684e5def2e9b59bba18a11644f9088c1ed6e4eb6e29c" Oct 30 21:33:09.530630 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:33:09.530611 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cd4b0b2f6294f76ef36684e5def2e9b59bba18a11644f9088c1ed6e4eb6e29c\": container with ID starting with 7cd4b0b2f6294f76ef36684e5def2e9b59bba18a11644f9088c1ed6e4eb6e29c not found: ID does not exist" containerID="7cd4b0b2f6294f76ef36684e5def2e9b59bba18a11644f9088c1ed6e4eb6e29c" Oct 30 21:33:09.530689 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:09.530639 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cd4b0b2f6294f76ef36684e5def2e9b59bba18a11644f9088c1ed6e4eb6e29c"} err="failed to get container status \"7cd4b0b2f6294f76ef36684e5def2e9b59bba18a11644f9088c1ed6e4eb6e29c\": rpc error: code = NotFound desc = could not find container \"7cd4b0b2f6294f76ef36684e5def2e9b59bba18a11644f9088c1ed6e4eb6e29c\": container with ID starting with 7cd4b0b2f6294f76ef36684e5def2e9b59bba18a11644f9088c1ed6e4eb6e29c not found: ID does not exist" Oct 30 21:33:09.687776 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:09.687744 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xkqx5"] Oct 30 21:33:09.738172 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:09.738144 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xkqx5"] Oct 30 21:33:10.573779 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:10.573749 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afd6d37d-851e-4b57-9041-1409c864beae" path="/var/lib/kubelet/pods/afd6d37d-851e-4b57-9041-1409c864beae/volumes" Oct 30 21:33:19.683202 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:33:19.683167 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:33:32.472721 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:32.472680 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:33:43.907894 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:43.907857 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:33:49.683520 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:33:49.683414 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:33:57.679078 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:57.678987 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl"] Oct 30 21:33:57.679550 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:57.679429 4560 cpu_manager.go:414] "containerMap: removing container" podUID="afd6d37d-851e-4b57-9041-1409c864beae" containerName="manager" Oct 30 21:33:57.679550 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:57.679447 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afd6d37d-851e-4b57-9041-1409c864beae" containerName="manager" Oct 30 21:33:57.679550 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:57.679528 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="afd6d37d-851e-4b57-9041-1409c864beae" containerName="manager" Oct 30 21:33:57.682222 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:57.682201 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" Oct 30 21:33:57.718251 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:57.718219 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e6e5570-8209-4ff2-84d2-1bf88ca77c81-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-zfpbl\" (UID: \"2e6e5570-8209-4ff2-84d2-1bf88ca77c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" Oct 30 21:33:57.718397 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:57.718312 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e6e5570-8209-4ff2-84d2-1bf88ca77c81-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-zfpbl\" (UID: \"2e6e5570-8209-4ff2-84d2-1bf88ca77c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" Oct 30 21:33:57.718397 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:57.718371 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e6e5570-8209-4ff2-84d2-1bf88ca77c81-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-zfpbl\" (UID: \"2e6e5570-8209-4ff2-84d2-1bf88ca77c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" Oct 30 21:33:57.718507 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:57.718409 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e6e5570-8209-4ff2-84d2-1bf88ca77c81-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-zfpbl\" (UID: \"2e6e5570-8209-4ff2-84d2-1bf88ca77c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" Oct 30 21:33:57.718563 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:57.718542 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e6e5570-8209-4ff2-84d2-1bf88ca77c81-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-zfpbl\" (UID: \"2e6e5570-8209-4ff2-84d2-1bf88ca77c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" Oct 30 21:33:57.718617 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:57.718575 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsc9h\" (UniqueName: \"kubernetes.io/projected/2e6e5570-8209-4ff2-84d2-1bf88ca77c81-kube-api-access-hsc9h\") pod \"maas-default-gateway-openshift-default-67bf986f99-zfpbl\" (UID: \"2e6e5570-8209-4ff2-84d2-1bf88ca77c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" Oct 30 21:33:57.718695 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:57.718678 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e6e5570-8209-4ff2-84d2-1bf88ca77c81-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-zfpbl\" (UID: \"2e6e5570-8209-4ff2-84d2-1bf88ca77c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" Oct 30 21:33:57.718732 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:57.718718 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e6e5570-8209-4ff2-84d2-1bf88ca77c81-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-zfpbl\" (UID: \"2e6e5570-8209-4ff2-84d2-1bf88ca77c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" Oct 30 21:33:57.718778 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:57.718752 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e6e5570-8209-4ff2-84d2-1bf88ca77c81-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-zfpbl\" (UID: \"2e6e5570-8209-4ff2-84d2-1bf88ca77c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" Oct 30 21:33:57.786432 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:57.786399 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl"] Oct 30 21:33:57.815795 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:57.815770 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-z4vl5\"" Oct 30 21:33:57.820081 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:57.820061 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e6e5570-8209-4ff2-84d2-1bf88ca77c81-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-zfpbl\" (UID: \"2e6e5570-8209-4ff2-84d2-1bf88ca77c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" Oct 30 21:33:57.820148 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:57.820095 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e6e5570-8209-4ff2-84d2-1bf88ca77c81-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-zfpbl\" (UID: \"2e6e5570-8209-4ff2-84d2-1bf88ca77c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" Oct 30 21:33:57.820148 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:57.820119 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e6e5570-8209-4ff2-84d2-1bf88ca77c81-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-zfpbl\" (UID: \"2e6e5570-8209-4ff2-84d2-1bf88ca77c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" Oct 30 21:33:57.820148 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:57.820137 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e6e5570-8209-4ff2-84d2-1bf88ca77c81-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-zfpbl\" (UID: \"2e6e5570-8209-4ff2-84d2-1bf88ca77c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" Oct 30 21:33:57.820293 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:57.820164 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e6e5570-8209-4ff2-84d2-1bf88ca77c81-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-zfpbl\" (UID: \"2e6e5570-8209-4ff2-84d2-1bf88ca77c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" Oct 30 21:33:57.820293 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:57.820186 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e6e5570-8209-4ff2-84d2-1bf88ca77c81-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-zfpbl\" (UID: \"2e6e5570-8209-4ff2-84d2-1bf88ca77c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" Oct 30 21:33:57.820293 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:57.820222 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e6e5570-8209-4ff2-84d2-1bf88ca77c81-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-zfpbl\" (UID: \"2e6e5570-8209-4ff2-84d2-1bf88ca77c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" Oct 30 21:33:57.820293 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:57.820274 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e6e5570-8209-4ff2-84d2-1bf88ca77c81-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-zfpbl\" (UID: \"2e6e5570-8209-4ff2-84d2-1bf88ca77c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" Oct 30 21:33:57.820518 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:57.820298 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsc9h\" (UniqueName: \"kubernetes.io/projected/2e6e5570-8209-4ff2-84d2-1bf88ca77c81-kube-api-access-hsc9h\") pod \"maas-default-gateway-openshift-default-67bf986f99-zfpbl\" (UID: \"2e6e5570-8209-4ff2-84d2-1bf88ca77c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" Oct 30 21:33:57.820634 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:57.820605 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e6e5570-8209-4ff2-84d2-1bf88ca77c81-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-zfpbl\" (UID: \"2e6e5570-8209-4ff2-84d2-1bf88ca77c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" Oct 30 21:33:57.820634 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:57.820615 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e6e5570-8209-4ff2-84d2-1bf88ca77c81-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-zfpbl\" (UID: \"2e6e5570-8209-4ff2-84d2-1bf88ca77c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" Oct 30 21:33:57.820785 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:57.820762 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e6e5570-8209-4ff2-84d2-1bf88ca77c81-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-zfpbl\" (UID: \"2e6e5570-8209-4ff2-84d2-1bf88ca77c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" Oct 30 21:33:57.820880 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:57.820863 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e6e5570-8209-4ff2-84d2-1bf88ca77c81-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-zfpbl\" (UID: \"2e6e5570-8209-4ff2-84d2-1bf88ca77c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" Oct 30 21:33:57.821012 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:57.820992 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e6e5570-8209-4ff2-84d2-1bf88ca77c81-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-zfpbl\" (UID: \"2e6e5570-8209-4ff2-84d2-1bf88ca77c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" Oct 30 21:33:57.822433 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:57.822406 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e6e5570-8209-4ff2-84d2-1bf88ca77c81-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-zfpbl\" (UID: \"2e6e5570-8209-4ff2-84d2-1bf88ca77c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" Oct 30 21:33:57.822890 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:57.822872 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e6e5570-8209-4ff2-84d2-1bf88ca77c81-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-zfpbl\" (UID: \"2e6e5570-8209-4ff2-84d2-1bf88ca77c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" Oct 30 21:33:57.881770 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:57.881736 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e6e5570-8209-4ff2-84d2-1bf88ca77c81-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-zfpbl\" (UID: \"2e6e5570-8209-4ff2-84d2-1bf88ca77c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" Oct 30 21:33:57.881873 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:57.881821 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsc9h\" (UniqueName: \"kubernetes.io/projected/2e6e5570-8209-4ff2-84d2-1bf88ca77c81-kube-api-access-hsc9h\") pod \"maas-default-gateway-openshift-default-67bf986f99-zfpbl\" (UID: \"2e6e5570-8209-4ff2-84d2-1bf88ca77c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" Oct 30 21:33:57.994950 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:57.994925 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" Oct 30 21:33:58.420695 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:58.420620 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl"] Oct 30 21:33:58.423944 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:33:58.423914 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e6e5570_8209_4ff2_84d2_1bf88ca77c81.slice/crio-a255d42ee9b6d16932ac395b15e2088b58a5e8a28597d5cde06d9922e16c14b8 WatchSource:0}: Error finding container a255d42ee9b6d16932ac395b15e2088b58a5e8a28597d5cde06d9922e16c14b8: Status 404 returned error can't find the container with id a255d42ee9b6d16932ac395b15e2088b58a5e8a28597d5cde06d9922e16c14b8 Oct 30 21:33:58.705450 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:33:58.705372 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" event={"ID":"2e6e5570-8209-4ff2-84d2-1bf88ca77c81","Type":"ContainerStarted","Data":"a255d42ee9b6d16932ac395b15e2088b58a5e8a28597d5cde06d9922e16c14b8"} Oct 30 21:34:00.999680 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:34:00.999641 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:34:02.307152 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:34:02.307106 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:34:02.307550 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:34:02.307191 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:34:02.307550 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:34:02.307239 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:34:02.722920 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:34:02.722832 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" event={"ID":"2e6e5570-8209-4ff2-84d2-1bf88ca77c81","Type":"ContainerStarted","Data":"f9ced8ce7beff670c25f059595d9f97e11d2bdb24b1ca6ba8b448593e32a3215"} Oct 30 21:34:02.895690 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:34:02.895633 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" podStartSLOduration=2.015079376 podStartE2EDuration="5.895614867s" totalImagesPullingTime="3.880535491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:02.895438677 +0000 UTC m=+1218.450138455" watchObservedRunningTime="2026-10-30 21:34:02.895614867 +0000 UTC m=+1218.450314644" Oct 30 21:34:02.995941 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:34:02.995909 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" Oct 30 21:34:03.996876 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:34:03.996840 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-zfpbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Oct 30 21:34:03.997258 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:34:03.996892 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" podUID="2e6e5570-8209-4ff2-84d2-1bf88ca77c81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Oct 30 21:34:04.995363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:34:04.995306 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-zfpbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Oct 30 21:34:04.995536 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:34:04.995391 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" podUID="2e6e5570-8209-4ff2-84d2-1bf88ca77c81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Oct 30 21:34:05.996160 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:34:05.996126 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-zfpbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Oct 30 21:34:05.996565 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:34:05.996177 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" podUID="2e6e5570-8209-4ff2-84d2-1bf88ca77c81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Oct 30 21:34:06.996288 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:34:06.996252 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-zfpbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Oct 30 21:34:06.996843 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:34:06.996311 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" podUID="2e6e5570-8209-4ff2-84d2-1bf88ca77c81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Oct 30 21:34:07.996327 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:34:07.996292 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-zfpbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Oct 30 21:34:07.996742 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:34:07.996362 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" podUID="2e6e5570-8209-4ff2-84d2-1bf88ca77c81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Oct 30 21:34:08.996181 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:34:08.996148 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-zfpbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Oct 30 21:34:08.996394 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:34:08.996203 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" podUID="2e6e5570-8209-4ff2-84d2-1bf88ca77c81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Oct 30 21:34:10.704672 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:34:10.704636 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" Oct 30 21:34:10.705279 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:34:10.704914 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" Oct 30 21:34:10.706278 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:34:10.706245 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" Oct 30 21:34:19.684170 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:34:19.684134 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:34:49.683843 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:34:49.683808 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:35:19.683201 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:35:19.683163 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:35:44.103183 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:44.103143 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-afbe5b-646894cd95-h5tqj"] Oct 30 21:35:44.105439 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:44.105422 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-afbe5b-646894cd95-h5tqj" Oct 30 21:35:44.201024 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:44.200995 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-afbe5b\"" Oct 30 21:35:44.201196 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:44.201076 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-afbe5b-dockercfg-vmksq\"" Oct 30 21:35:44.202450 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:44.202423 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-afbe5b-646894cd95-h5tqj"] Oct 30 21:35:44.215903 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:44.215871 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1807c151-3102-4158-b028-d7dd697f4ad6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-afbe5b-646894cd95-h5tqj\" (UID: \"1807c151-3102-4158-b028-d7dd697f4ad6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-afbe5b-646894cd95-h5tqj" Oct 30 21:35:44.216096 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:44.215927 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rdr5\" (UniqueName: \"kubernetes.io/projected/1807c151-3102-4158-b028-d7dd697f4ad6-kube-api-access-7rdr5\") pod \"payload-pre-processing-e2e-worker-w0-afbe5b-646894cd95-h5tqj\" (UID: \"1807c151-3102-4158-b028-d7dd697f4ad6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-afbe5b-646894cd95-h5tqj" Oct 30 21:35:44.257097 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:44.257066 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-afbe5b-85d55db988-8c4wd"] Oct 30 21:35:44.259481 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:44.259464 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-afbe5b-85d55db988-8c4wd" Oct 30 21:35:44.316156 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:44.316110 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-afbe5b-85d55db988-8c4wd"] Oct 30 21:35:44.316578 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:44.316551 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1807c151-3102-4158-b028-d7dd697f4ad6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-afbe5b-646894cd95-h5tqj\" (UID: \"1807c151-3102-4158-b028-d7dd697f4ad6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-afbe5b-646894cd95-h5tqj" Oct 30 21:35:44.316672 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:44.316600 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/024e6f83-289f-499a-be2d-5aed8a5a6c6d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-afbe5b-85d55db988-8c4wd\" (UID: \"024e6f83-289f-499a-be2d-5aed8a5a6c6d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-afbe5b-85d55db988-8c4wd" Oct 30 21:35:44.316672 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:44.316632 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rdr5\" (UniqueName: \"kubernetes.io/projected/1807c151-3102-4158-b028-d7dd697f4ad6-kube-api-access-7rdr5\") pod \"payload-pre-processing-e2e-worker-w0-afbe5b-646894cd95-h5tqj\" (UID: \"1807c151-3102-4158-b028-d7dd697f4ad6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-afbe5b-646894cd95-h5tqj" Oct 30 21:35:44.316672 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:44.316651 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57t2t\" (UniqueName: \"kubernetes.io/projected/024e6f83-289f-499a-be2d-5aed8a5a6c6d-kube-api-access-57t2t\") pod \"payload-processing-e2e-worker-w0-afbe5b-85d55db988-8c4wd\" (UID: \"024e6f83-289f-499a-be2d-5aed8a5a6c6d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-afbe5b-85d55db988-8c4wd" Oct 30 21:35:44.317746 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:44.317721 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1807c151-3102-4158-b028-d7dd697f4ad6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-afbe5b-646894cd95-h5tqj\" (UID: \"1807c151-3102-4158-b028-d7dd697f4ad6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-afbe5b-646894cd95-h5tqj" Oct 30 21:35:44.375877 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:44.375803 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rdr5\" (UniqueName: \"kubernetes.io/projected/1807c151-3102-4158-b028-d7dd697f4ad6-kube-api-access-7rdr5\") pod \"payload-pre-processing-e2e-worker-w0-afbe5b-646894cd95-h5tqj\" (UID: \"1807c151-3102-4158-b028-d7dd697f4ad6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-afbe5b-646894cd95-h5tqj" Oct 30 21:35:44.417255 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:44.417220 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-afbe5b-646894cd95-h5tqj" Oct 30 21:35:44.417435 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:44.417417 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57t2t\" (UniqueName: \"kubernetes.io/projected/024e6f83-289f-499a-be2d-5aed8a5a6c6d-kube-api-access-57t2t\") pod \"payload-processing-e2e-worker-w0-afbe5b-85d55db988-8c4wd\" (UID: \"024e6f83-289f-499a-be2d-5aed8a5a6c6d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-afbe5b-85d55db988-8c4wd" Oct 30 21:35:44.417532 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:44.417518 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/024e6f83-289f-499a-be2d-5aed8a5a6c6d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-afbe5b-85d55db988-8c4wd\" (UID: \"024e6f83-289f-499a-be2d-5aed8a5a6c6d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-afbe5b-85d55db988-8c4wd" Oct 30 21:35:44.418146 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:44.418127 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/024e6f83-289f-499a-be2d-5aed8a5a6c6d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-afbe5b-85d55db988-8c4wd\" (UID: \"024e6f83-289f-499a-be2d-5aed8a5a6c6d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-afbe5b-85d55db988-8c4wd" Oct 30 21:35:44.475956 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:44.475925 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57t2t\" (UniqueName: \"kubernetes.io/projected/024e6f83-289f-499a-be2d-5aed8a5a6c6d-kube-api-access-57t2t\") pod \"payload-processing-e2e-worker-w0-afbe5b-85d55db988-8c4wd\" (UID: \"024e6f83-289f-499a-be2d-5aed8a5a6c6d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-afbe5b-85d55db988-8c4wd" Oct 30 21:35:44.570826 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:44.570796 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-afbe5b-85d55db988-8c4wd" Oct 30 21:35:44.798479 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:44.798441 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-afbe5b-646894cd95-h5tqj"] Oct 30 21:35:44.806475 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:35:44.806439 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1807c151_3102_4158_b028_d7dd697f4ad6.slice/crio-14a381763658f8b3da22bfc2e935840002b561a880c6bacb4e763832f25a10a8 WatchSource:0}: Error finding container 14a381763658f8b3da22bfc2e935840002b561a880c6bacb4e763832f25a10a8: Status 404 returned error can't find the container with id 14a381763658f8b3da22bfc2e935840002b561a880c6bacb4e763832f25a10a8 Oct 30 21:35:44.941812 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:44.941727 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-afbe5b-85d55db988-8c4wd"] Oct 30 21:35:44.948886 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:35:44.948857 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod024e6f83_289f_499a_be2d_5aed8a5a6c6d.slice/crio-5af9476a3d41ed91cdd79a4fc681c9362bd7bfe6b5c619ad0404dbea0e089197 WatchSource:0}: Error finding container 5af9476a3d41ed91cdd79a4fc681c9362bd7bfe6b5c619ad0404dbea0e089197: Status 404 returned error can't find the container with id 5af9476a3d41ed91cdd79a4fc681c9362bd7bfe6b5c619ad0404dbea0e089197 Oct 30 21:35:45.139774 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:45.139734 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-afbe5b-85d55db988-8c4wd" event={"ID":"024e6f83-289f-499a-be2d-5aed8a5a6c6d","Type":"ContainerStarted","Data":"5af9476a3d41ed91cdd79a4fc681c9362bd7bfe6b5c619ad0404dbea0e089197"} Oct 30 21:35:45.140864 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:45.140841 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-afbe5b-646894cd95-h5tqj" event={"ID":"1807c151-3102-4158-b028-d7dd697f4ad6","Type":"ContainerStarted","Data":"14a381763658f8b3da22bfc2e935840002b561a880c6bacb4e763832f25a10a8"} Oct 30 21:35:45.901095 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:45.901060 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:35:48.157128 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:48.157025 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-afbe5b-85d55db988-8c4wd" event={"ID":"024e6f83-289f-499a-be2d-5aed8a5a6c6d","Type":"ContainerStarted","Data":"2660ad319e7c9efa9255615da5d042b7aa05f14a25ba1091e2ae4464ba32134a"} Oct 30 21:35:48.157128 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:48.157091 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-afbe5b-85d55db988-8c4wd" Oct 30 21:35:48.158519 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:48.158493 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-afbe5b-646894cd95-h5tqj" event={"ID":"1807c151-3102-4158-b028-d7dd697f4ad6","Type":"ContainerStarted","Data":"6571f2bf6dceb7f69dac253f0b49495f4b43dd0f9035ebec675e8bed5206716f"} Oct 30 21:35:48.158648 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:48.158634 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-afbe5b-646894cd95-h5tqj" Oct 30 21:35:48.267837 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:48.267777 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-afbe5b-85d55db988-8c4wd" podStartSLOduration=1.2580224819999999 podStartE2EDuration="4.267759838s" totalImagesPullingTime="3.009737356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:48.2672218 +0000 UTC m=+1323.821921576" watchObservedRunningTime="2026-10-30 21:35:48.267759838 +0000 UTC m=+1323.822459616" Oct 30 21:35:49.467057 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:49.466988 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-afbe5b-646894cd95-h5tqj" podStartSLOduration=2.431432427 podStartE2EDuration="5.466971253s" totalImagesPullingTime="3.035538826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:48.38696001 +0000 UTC m=+1323.941659787" watchObservedRunningTime="2026-10-30 21:35:49.466971253 +0000 UTC m=+1325.021671031" Oct 30 21:35:49.467757 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:49.467736 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a2dced-7f7d56769b-9fh9l"] Oct 30 21:35:49.471487 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:49.471471 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-a2dced-7f7d56769b-9fh9l" Oct 30 21:35:49.520590 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:49.520563 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-a2dced\"" Oct 30 21:35:49.520590 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:49.520585 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-a2dced-dockercfg-nnxzr\"" Oct 30 21:35:49.567011 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:49.566975 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55bca164-d7e2-4f16-a24d-2aea919e3047-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-a2dced-7f7d56769b-9fh9l\" (UID: \"55bca164-d7e2-4f16-a24d-2aea919e3047\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a2dced-7f7d56769b-9fh9l" Oct 30 21:35:49.567168 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:49.567032 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p9g4\" (UniqueName: \"kubernetes.io/projected/55bca164-d7e2-4f16-a24d-2aea919e3047-kube-api-access-9p9g4\") pod \"payload-processing-e2e-worker-w6-a2dced-7f7d56769b-9fh9l\" (UID: \"55bca164-d7e2-4f16-a24d-2aea919e3047\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a2dced-7f7d56769b-9fh9l" Oct 30 21:35:49.568161 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:49.568134 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a2dced-7f7d56769b-9fh9l"] Oct 30 21:35:49.667975 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:49.667944 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55bca164-d7e2-4f16-a24d-2aea919e3047-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-a2dced-7f7d56769b-9fh9l\" (UID: \"55bca164-d7e2-4f16-a24d-2aea919e3047\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a2dced-7f7d56769b-9fh9l" Oct 30 21:35:49.668127 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:49.668000 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p9g4\" (UniqueName: \"kubernetes.io/projected/55bca164-d7e2-4f16-a24d-2aea919e3047-kube-api-access-9p9g4\") pod \"payload-processing-e2e-worker-w6-a2dced-7f7d56769b-9fh9l\" (UID: \"55bca164-d7e2-4f16-a24d-2aea919e3047\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a2dced-7f7d56769b-9fh9l" Oct 30 21:35:49.668645 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:49.668627 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55bca164-d7e2-4f16-a24d-2aea919e3047-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-a2dced-7f7d56769b-9fh9l\" (UID: \"55bca164-d7e2-4f16-a24d-2aea919e3047\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a2dced-7f7d56769b-9fh9l" Oct 30 21:35:49.684711 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:35:49.684671 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:35:49.725866 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:49.725845 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p9g4\" (UniqueName: \"kubernetes.io/projected/55bca164-d7e2-4f16-a24d-2aea919e3047-kube-api-access-9p9g4\") pod \"payload-processing-e2e-worker-w6-a2dced-7f7d56769b-9fh9l\" (UID: \"55bca164-d7e2-4f16-a24d-2aea919e3047\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a2dced-7f7d56769b-9fh9l" Oct 30 21:35:49.783112 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:49.783085 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-a2dced-7f7d56769b-9fh9l" Oct 30 21:35:50.153533 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:50.153457 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a2dced-7f7d56769b-9fh9l"] Oct 30 21:35:50.161043 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:35:50.161015 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55bca164_d7e2_4f16_a24d_2aea919e3047.slice/crio-3a3564b78e707edb7b321f704c9a43a605643540ebb60e252cd20aed89e51fe6 WatchSource:0}: Error finding container 3a3564b78e707edb7b321f704c9a43a605643540ebb60e252cd20aed89e51fe6: Status 404 returned error can't find the container with id 3a3564b78e707edb7b321f704c9a43a605643540ebb60e252cd20aed89e51fe6 Oct 30 21:35:50.168516 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:50.168484 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-a2dced-7f7d56769b-9fh9l" event={"ID":"55bca164-d7e2-4f16-a24d-2aea919e3047","Type":"ContainerStarted","Data":"3a3564b78e707edb7b321f704c9a43a605643540ebb60e252cd20aed89e51fe6"} Oct 30 21:35:51.174463 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:51.174424 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-a2dced-7f7d56769b-9fh9l" event={"ID":"55bca164-d7e2-4f16-a24d-2aea919e3047","Type":"ContainerStarted","Data":"e250887e902d002a0e7f619d1ad6d1ca7b6bb53a9b4da7b021a8332d57e30444"} Oct 30 21:35:51.174859 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:51.174502 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-a2dced-7f7d56769b-9fh9l" Oct 30 21:35:51.293460 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:51.293401 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-a2dced-7f7d56769b-9fh9l" podStartSLOduration=2.293387082 podStartE2EDuration="2.293387082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:51.292517518 +0000 UTC m=+1326.847217295" watchObservedRunningTime="2026-10-30 21:35:51.293387082 +0000 UTC m=+1326.848086875" Oct 30 21:35:55.230745 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:55.230707 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:35:59.165977 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:59.165949 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-afbe5b-646894cd95-h5tqj" Oct 30 21:35:59.166481 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:35:59.166013 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-afbe5b-85d55db988-8c4wd" Oct 30 21:36:02.182424 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:02.182393 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-a2dced-7f7d56769b-9fh9l" Oct 30 21:36:06.522267 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:06.522226 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:36:16.043149 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:16.043114 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk"] Oct 30 21:36:16.046312 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:16.046290 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" Oct 30 21:36:16.142357 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:16.142297 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-554f46\"/\"openshift-service-ca.crt\"" Oct 30 21:36:16.142357 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:16.142335 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-554f46\"/\"kube-root-ca.crt\"" Oct 30 21:36:16.142548 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:16.142426 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-554f46\"/\"default-dockercfg-dlzxb\"" Oct 30 21:36:16.142548 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:16.142515 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-554f46\"/\"simulator-w4-554f46-kserve-self-signed-certs\"" Oct 30 21:36:16.143982 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:16.143964 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk"] Oct 30 21:36:16.204126 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:16.203716 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxhn8\" (UniqueName: \"kubernetes.io/projected/0ff8ae26-4364-4c4e-9285-721d1a486a9b-kube-api-access-qxhn8\") pod \"simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk\" (UID: \"0ff8ae26-4364-4c4e-9285-721d1a486a9b\") " pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" Oct 30 21:36:16.204126 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:16.203820 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ff8ae26-4364-4c4e-9285-721d1a486a9b-dshm\") pod \"simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk\" (UID: \"0ff8ae26-4364-4c4e-9285-721d1a486a9b\") " pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" Oct 30 21:36:16.204126 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:16.203866 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ff8ae26-4364-4c4e-9285-721d1a486a9b-tmp-dir\") pod \"simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk\" (UID: \"0ff8ae26-4364-4c4e-9285-721d1a486a9b\") " pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" Oct 30 21:36:16.204126 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:16.203903 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ff8ae26-4364-4c4e-9285-721d1a486a9b-home\") pod \"simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk\" (UID: \"0ff8ae26-4364-4c4e-9285-721d1a486a9b\") " pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" Oct 30 21:36:16.204126 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:16.203950 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ff8ae26-4364-4c4e-9285-721d1a486a9b-tls-certs\") pod \"simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk\" (UID: \"0ff8ae26-4364-4c4e-9285-721d1a486a9b\") " pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" Oct 30 21:36:16.204126 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:16.203978 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ff8ae26-4364-4c4e-9285-721d1a486a9b-model-cache\") pod \"simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk\" (UID: \"0ff8ae26-4364-4c4e-9285-721d1a486a9b\") " pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" Oct 30 21:36:16.304784 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:16.304708 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ff8ae26-4364-4c4e-9285-721d1a486a9b-dshm\") pod \"simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk\" (UID: \"0ff8ae26-4364-4c4e-9285-721d1a486a9b\") " pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" Oct 30 21:36:16.304784 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:16.304748 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ff8ae26-4364-4c4e-9285-721d1a486a9b-tmp-dir\") pod \"simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk\" (UID: \"0ff8ae26-4364-4c4e-9285-721d1a486a9b\") " pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" Oct 30 21:36:16.304784 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:16.304769 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ff8ae26-4364-4c4e-9285-721d1a486a9b-home\") pod \"simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk\" (UID: \"0ff8ae26-4364-4c4e-9285-721d1a486a9b\") " pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" Oct 30 21:36:16.305062 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:16.304802 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ff8ae26-4364-4c4e-9285-721d1a486a9b-tls-certs\") pod \"simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk\" (UID: \"0ff8ae26-4364-4c4e-9285-721d1a486a9b\") " pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" Oct 30 21:36:16.305062 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:16.304820 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ff8ae26-4364-4c4e-9285-721d1a486a9b-model-cache\") pod \"simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk\" (UID: \"0ff8ae26-4364-4c4e-9285-721d1a486a9b\") " pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" Oct 30 21:36:16.305062 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:16.304839 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxhn8\" (UniqueName: \"kubernetes.io/projected/0ff8ae26-4364-4c4e-9285-721d1a486a9b-kube-api-access-qxhn8\") pod \"simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk\" (UID: \"0ff8ae26-4364-4c4e-9285-721d1a486a9b\") " pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" Oct 30 21:36:16.305248 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:16.305225 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ff8ae26-4364-4c4e-9285-721d1a486a9b-tmp-dir\") pod \"simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk\" (UID: \"0ff8ae26-4364-4c4e-9285-721d1a486a9b\") " pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" Oct 30 21:36:16.305301 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:16.305251 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ff8ae26-4364-4c4e-9285-721d1a486a9b-model-cache\") pod \"simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk\" (UID: \"0ff8ae26-4364-4c4e-9285-721d1a486a9b\") " pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" Oct 30 21:36:16.305301 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:16.305278 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ff8ae26-4364-4c4e-9285-721d1a486a9b-home\") pod \"simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk\" (UID: \"0ff8ae26-4364-4c4e-9285-721d1a486a9b\") " pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" Oct 30 21:36:16.307016 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:16.306988 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ff8ae26-4364-4c4e-9285-721d1a486a9b-dshm\") pod \"simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk\" (UID: \"0ff8ae26-4364-4c4e-9285-721d1a486a9b\") " pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" Oct 30 21:36:16.307234 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:16.307215 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ff8ae26-4364-4c4e-9285-721d1a486a9b-tls-certs\") pod \"simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk\" (UID: \"0ff8ae26-4364-4c4e-9285-721d1a486a9b\") " pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" Oct 30 21:36:16.363911 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:16.363883 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxhn8\" (UniqueName: \"kubernetes.io/projected/0ff8ae26-4364-4c4e-9285-721d1a486a9b-kube-api-access-qxhn8\") pod \"simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk\" (UID: \"0ff8ae26-4364-4c4e-9285-721d1a486a9b\") " pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" Oct 30 21:36:16.657112 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:16.657035 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" Oct 30 21:36:17.032096 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:17.032068 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk"] Oct 30 21:36:17.040019 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:36:17.039992 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ff8ae26_4364_4c4e_9285_721d1a486a9b.slice/crio-fca0e6af1fd8fc23e2487862fdb155524671e72eb2dc639e2ca6a01dddae7ca2 WatchSource:0}: Error finding container fca0e6af1fd8fc23e2487862fdb155524671e72eb2dc639e2ca6a01dddae7ca2: Status 404 returned error can't find the container with id fca0e6af1fd8fc23e2487862fdb155524671e72eb2dc639e2ca6a01dddae7ca2 Oct 30 21:36:17.042636 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:17.042615 4560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:36:17.289616 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:17.289535 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" event={"ID":"0ff8ae26-4364-4c4e-9285-721d1a486a9b","Type":"ContainerStarted","Data":"fca0e6af1fd8fc23e2487862fdb155524671e72eb2dc639e2ca6a01dddae7ca2"} Oct 30 21:36:18.012908 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:18.012876 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:36:19.300374 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:19.300319 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" event={"ID":"0ff8ae26-4364-4c4e-9285-721d1a486a9b","Type":"ContainerStarted","Data":"c1e96ba62d7d61acfa14242db5c74d9ccef1d0cab5ff66e0f1b244c548150fde"} Oct 30 21:36:19.411217 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:19.411141 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" podStartSLOduration=1.488954217 podStartE2EDuration="3.411126518s" totalImagesPullingTime="1.922172301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:19.410875544 +0000 UTC m=+1354.965575320" watchObservedRunningTime="2026-10-30 21:36:19.411126518 +0000 UTC m=+1354.965826304" Oct 30 21:36:19.658176 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:19.658097 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" Oct 30 21:36:19.683480 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:36:19.683449 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:36:26.658140 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:26.658099 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" Oct 30 21:36:26.671104 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:26.671070 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" Oct 30 21:36:27.344842 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:27.344816 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" Oct 30 21:36:29.300428 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:29.300391 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:36:41.536354 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:41.536303 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q"] Oct 30 21:36:41.539558 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:41.539530 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" Oct 30 21:36:41.599288 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:41.599238 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45a5bac0-28ab-4c78-856e-303f6d4522de-tls-certs\") pod \"simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q\" (UID: \"45a5bac0-28ab-4c78-856e-303f6d4522de\") " pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" Oct 30 21:36:41.599456 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:41.599375 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45a5bac0-28ab-4c78-856e-303f6d4522de-home\") pod \"simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q\" (UID: \"45a5bac0-28ab-4c78-856e-303f6d4522de\") " pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" Oct 30 21:36:41.599456 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:41.599412 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45a5bac0-28ab-4c78-856e-303f6d4522de-model-cache\") pod \"simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q\" (UID: \"45a5bac0-28ab-4c78-856e-303f6d4522de\") " pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" Oct 30 21:36:41.599575 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:41.599490 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45a5bac0-28ab-4c78-856e-303f6d4522de-tmp-dir\") pod \"simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q\" (UID: \"45a5bac0-28ab-4c78-856e-303f6d4522de\") " pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" Oct 30 21:36:41.599575 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:41.599516 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b7rp\" (UniqueName: \"kubernetes.io/projected/45a5bac0-28ab-4c78-856e-303f6d4522de-kube-api-access-8b7rp\") pod \"simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q\" (UID: \"45a5bac0-28ab-4c78-856e-303f6d4522de\") " pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" Oct 30 21:36:41.599575 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:41.599564 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45a5bac0-28ab-4c78-856e-303f6d4522de-dshm\") pod \"simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q\" (UID: \"45a5bac0-28ab-4c78-856e-303f6d4522de\") " pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" Oct 30 21:36:41.635275 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:41.635229 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-a2dced\"/\"kube-root-ca.crt\"" Oct 30 21:36:41.635275 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:41.635258 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-a2dced\"/\"openshift-service-ca.crt\"" Oct 30 21:36:41.635275 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:41.635229 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a2dced\"/\"default-dockercfg-smfxk\"" Oct 30 21:36:41.635508 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:41.635297 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a2dced\"/\"simulator-w6-a2dced-kserve-self-signed-certs\"" Oct 30 21:36:41.636708 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:41.636687 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q"] Oct 30 21:36:41.700579 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:41.700540 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45a5bac0-28ab-4c78-856e-303f6d4522de-tmp-dir\") pod \"simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q\" (UID: \"45a5bac0-28ab-4c78-856e-303f6d4522de\") " pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" Oct 30 21:36:41.700742 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:41.700591 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b7rp\" (UniqueName: \"kubernetes.io/projected/45a5bac0-28ab-4c78-856e-303f6d4522de-kube-api-access-8b7rp\") pod \"simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q\" (UID: \"45a5bac0-28ab-4c78-856e-303f6d4522de\") " pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" Oct 30 21:36:41.700742 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:41.700639 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45a5bac0-28ab-4c78-856e-303f6d4522de-dshm\") pod \"simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q\" (UID: \"45a5bac0-28ab-4c78-856e-303f6d4522de\") " pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" Oct 30 21:36:41.700742 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:41.700693 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45a5bac0-28ab-4c78-856e-303f6d4522de-tls-certs\") pod \"simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q\" (UID: \"45a5bac0-28ab-4c78-856e-303f6d4522de\") " pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" Oct 30 21:36:41.700742 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:41.700733 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45a5bac0-28ab-4c78-856e-303f6d4522de-home\") pod \"simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q\" (UID: \"45a5bac0-28ab-4c78-856e-303f6d4522de\") " pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" Oct 30 21:36:41.700999 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:41.700755 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45a5bac0-28ab-4c78-856e-303f6d4522de-model-cache\") pod \"simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q\" (UID: \"45a5bac0-28ab-4c78-856e-303f6d4522de\") " pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" Oct 30 21:36:41.700999 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:41.700993 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45a5bac0-28ab-4c78-856e-303f6d4522de-tmp-dir\") pod \"simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q\" (UID: \"45a5bac0-28ab-4c78-856e-303f6d4522de\") " pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" Oct 30 21:36:41.701115 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:41.701097 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45a5bac0-28ab-4c78-856e-303f6d4522de-model-cache\") pod \"simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q\" (UID: \"45a5bac0-28ab-4c78-856e-303f6d4522de\") " pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" Oct 30 21:36:41.701163 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:41.701147 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45a5bac0-28ab-4c78-856e-303f6d4522de-home\") pod \"simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q\" (UID: \"45a5bac0-28ab-4c78-856e-303f6d4522de\") " pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" Oct 30 21:36:41.703014 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:41.702993 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45a5bac0-28ab-4c78-856e-303f6d4522de-dshm\") pod \"simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q\" (UID: \"45a5bac0-28ab-4c78-856e-303f6d4522de\") " pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" Oct 30 21:36:41.703113 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:41.703081 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45a5bac0-28ab-4c78-856e-303f6d4522de-tls-certs\") pod \"simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q\" (UID: \"45a5bac0-28ab-4c78-856e-303f6d4522de\") " pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" Oct 30 21:36:41.758447 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:41.758413 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b7rp\" (UniqueName: \"kubernetes.io/projected/45a5bac0-28ab-4c78-856e-303f6d4522de-kube-api-access-8b7rp\") pod \"simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q\" (UID: \"45a5bac0-28ab-4c78-856e-303f6d4522de\") " pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" Oct 30 21:36:41.850528 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:41.850452 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" Oct 30 21:36:42.229908 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:42.229873 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q"] Oct 30 21:36:42.236711 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:36:42.236688 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45a5bac0_28ab_4c78_856e_303f6d4522de.slice/crio-9b550b68cb3708034bbcb41d70ed45de4aaf4116ed0081909ebd3c8e17f16aa8 WatchSource:0}: Error finding container 9b550b68cb3708034bbcb41d70ed45de4aaf4116ed0081909ebd3c8e17f16aa8: Status 404 returned error can't find the container with id 9b550b68cb3708034bbcb41d70ed45de4aaf4116ed0081909ebd3c8e17f16aa8 Oct 30 21:36:42.396065 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:42.396015 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" event={"ID":"45a5bac0-28ab-4c78-856e-303f6d4522de","Type":"ContainerStarted","Data":"0b9c22a19015ef176dc9de336fdbb6ad4c24de790893208a6d6bd106da2ff3ed"} Oct 30 21:36:42.396225 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:42.396079 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" event={"ID":"45a5bac0-28ab-4c78-856e-303f6d4522de","Type":"ContainerStarted","Data":"9b550b68cb3708034bbcb41d70ed45de4aaf4116ed0081909ebd3c8e17f16aa8"} Oct 30 21:36:42.512593 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:42.512497 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" podStartSLOduration=1.5124836130000001 podStartE2EDuration="1.512483613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:42.512277772 +0000 UTC m=+1378.066977553" watchObservedRunningTime="2026-10-30 21:36:42.512483613 +0000 UTC m=+1378.067183389" Oct 30 21:36:42.850624 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:42.850544 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" Oct 30 21:36:49.531300 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:49.531264 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:36:49.685442 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:36:49.685402 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:36:51.850946 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:51.850905 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" Oct 30 21:36:51.863444 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:51.863420 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" Oct 30 21:36:52.446960 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:52.446931 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" Oct 30 21:36:57.104378 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:57.104288 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm"] Oct 30 21:36:57.110292 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:57.110273 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" Oct 30 21:36:57.132857 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:57.132829 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27df5dfe-47a4-4ebb-bd09-1ceef406600a-dshm\") pod \"simulator-w1-19dd4f-kserve-674558b488-b4pmm\" (UID: \"27df5dfe-47a4-4ebb-bd09-1ceef406600a\") " pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" Oct 30 21:36:57.132979 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:57.132874 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27df5dfe-47a4-4ebb-bd09-1ceef406600a-model-cache\") pod \"simulator-w1-19dd4f-kserve-674558b488-b4pmm\" (UID: \"27df5dfe-47a4-4ebb-bd09-1ceef406600a\") " pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" Oct 30 21:36:57.132979 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:57.132957 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27df5dfe-47a4-4ebb-bd09-1ceef406600a-home\") pod \"simulator-w1-19dd4f-kserve-674558b488-b4pmm\" (UID: \"27df5dfe-47a4-4ebb-bd09-1ceef406600a\") " pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" Oct 30 21:36:57.133065 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:57.133013 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27df5dfe-47a4-4ebb-bd09-1ceef406600a-tmp-dir\") pod \"simulator-w1-19dd4f-kserve-674558b488-b4pmm\" (UID: \"27df5dfe-47a4-4ebb-bd09-1ceef406600a\") " pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" Oct 30 21:36:57.133065 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:57.133031 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27df5dfe-47a4-4ebb-bd09-1ceef406600a-tls-certs\") pod \"simulator-w1-19dd4f-kserve-674558b488-b4pmm\" (UID: \"27df5dfe-47a4-4ebb-bd09-1ceef406600a\") " pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" Oct 30 21:36:57.133141 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:57.133120 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5znbk\" (UniqueName: \"kubernetes.io/projected/27df5dfe-47a4-4ebb-bd09-1ceef406600a-kube-api-access-5znbk\") pod \"simulator-w1-19dd4f-kserve-674558b488-b4pmm\" (UID: \"27df5dfe-47a4-4ebb-bd09-1ceef406600a\") " pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" Oct 30 21:36:57.207838 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:57.207805 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-19dd4f\"/\"default-dockercfg-w6f95\"" Oct 30 21:36:57.208206 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:57.208184 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-19dd4f\"/\"simulator-w1-19dd4f-kserve-self-signed-certs\"" Oct 30 21:36:57.209578 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:57.209558 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm"] Oct 30 21:36:57.222753 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:57.222734 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-19dd4f\"/\"kube-root-ca.crt\"" Oct 30 21:36:57.222753 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:57.222750 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-19dd4f\"/\"openshift-service-ca.crt\"" Oct 30 21:36:57.234100 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:57.234081 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27df5dfe-47a4-4ebb-bd09-1ceef406600a-dshm\") pod \"simulator-w1-19dd4f-kserve-674558b488-b4pmm\" (UID: \"27df5dfe-47a4-4ebb-bd09-1ceef406600a\") " pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" Oct 30 21:36:57.234175 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:57.234110 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27df5dfe-47a4-4ebb-bd09-1ceef406600a-model-cache\") pod \"simulator-w1-19dd4f-kserve-674558b488-b4pmm\" (UID: \"27df5dfe-47a4-4ebb-bd09-1ceef406600a\") " pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" Oct 30 21:36:57.234175 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:57.234136 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27df5dfe-47a4-4ebb-bd09-1ceef406600a-home\") pod \"simulator-w1-19dd4f-kserve-674558b488-b4pmm\" (UID: \"27df5dfe-47a4-4ebb-bd09-1ceef406600a\") " pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" Oct 30 21:36:57.234175 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:57.234174 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27df5dfe-47a4-4ebb-bd09-1ceef406600a-tmp-dir\") pod \"simulator-w1-19dd4f-kserve-674558b488-b4pmm\" (UID: \"27df5dfe-47a4-4ebb-bd09-1ceef406600a\") " pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" Oct 30 21:36:57.234326 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:57.234190 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27df5dfe-47a4-4ebb-bd09-1ceef406600a-tls-certs\") pod \"simulator-w1-19dd4f-kserve-674558b488-b4pmm\" (UID: \"27df5dfe-47a4-4ebb-bd09-1ceef406600a\") " pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" Oct 30 21:36:57.234326 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:57.234219 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5znbk\" (UniqueName: \"kubernetes.io/projected/27df5dfe-47a4-4ebb-bd09-1ceef406600a-kube-api-access-5znbk\") pod \"simulator-w1-19dd4f-kserve-674558b488-b4pmm\" (UID: \"27df5dfe-47a4-4ebb-bd09-1ceef406600a\") " pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" Oct 30 21:36:57.234538 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:57.234520 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27df5dfe-47a4-4ebb-bd09-1ceef406600a-model-cache\") pod \"simulator-w1-19dd4f-kserve-674558b488-b4pmm\" (UID: \"27df5dfe-47a4-4ebb-bd09-1ceef406600a\") " pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" Oct 30 21:36:57.234633 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:57.234598 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27df5dfe-47a4-4ebb-bd09-1ceef406600a-home\") pod \"simulator-w1-19dd4f-kserve-674558b488-b4pmm\" (UID: \"27df5dfe-47a4-4ebb-bd09-1ceef406600a\") " pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" Oct 30 21:36:57.234742 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:57.234635 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27df5dfe-47a4-4ebb-bd09-1ceef406600a-tmp-dir\") pod \"simulator-w1-19dd4f-kserve-674558b488-b4pmm\" (UID: \"27df5dfe-47a4-4ebb-bd09-1ceef406600a\") " pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" Oct 30 21:36:57.236467 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:57.236446 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27df5dfe-47a4-4ebb-bd09-1ceef406600a-dshm\") pod \"simulator-w1-19dd4f-kserve-674558b488-b4pmm\" (UID: \"27df5dfe-47a4-4ebb-bd09-1ceef406600a\") " pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" Oct 30 21:36:57.236741 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:57.236721 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27df5dfe-47a4-4ebb-bd09-1ceef406600a-tls-certs\") pod \"simulator-w1-19dd4f-kserve-674558b488-b4pmm\" (UID: \"27df5dfe-47a4-4ebb-bd09-1ceef406600a\") " pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" Oct 30 21:36:57.292300 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:57.292262 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5znbk\" (UniqueName: \"kubernetes.io/projected/27df5dfe-47a4-4ebb-bd09-1ceef406600a-kube-api-access-5znbk\") pod \"simulator-w1-19dd4f-kserve-674558b488-b4pmm\" (UID: \"27df5dfe-47a4-4ebb-bd09-1ceef406600a\") " pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" Oct 30 21:36:57.424615 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:57.424536 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" Oct 30 21:36:57.803935 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:57.803900 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm"] Oct 30 21:36:57.810701 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:36:57.810672 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27df5dfe_47a4_4ebb_bd09_1ceef406600a.slice/crio-a08068f3131a903b082fcbb9720579569ecc4feefa6aa74c0e1d70dac4d77c2e WatchSource:0}: Error finding container a08068f3131a903b082fcbb9720579569ecc4feefa6aa74c0e1d70dac4d77c2e: Status 404 returned error can't find the container with id a08068f3131a903b082fcbb9720579569ecc4feefa6aa74c0e1d70dac4d77c2e Oct 30 21:36:58.462734 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:58.462699 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" event={"ID":"27df5dfe-47a4-4ebb-bd09-1ceef406600a","Type":"ContainerStarted","Data":"ca401edad43530267514279db754aebfe262498a484c0261fe962e42b5a9a1a2"} Oct 30 21:36:58.462734 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:58.462741 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" event={"ID":"27df5dfe-47a4-4ebb-bd09-1ceef406600a","Type":"ContainerStarted","Data":"a08068f3131a903b082fcbb9720579569ecc4feefa6aa74c0e1d70dac4d77c2e"} Oct 30 21:36:58.574998 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:58.574918 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" podStartSLOduration=1.574901616 podStartE2EDuration="1.574901616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:58.574207995 +0000 UTC m=+1394.128907771" watchObservedRunningTime="2026-10-30 21:36:58.574901616 +0000 UTC m=+1394.129601394" Oct 30 21:36:59.425405 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:36:59.425370 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" Oct 30 21:37:04.577857 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:04.577821 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:37:07.424833 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:07.424795 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" Oct 30 21:37:07.437442 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:07.437415 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" Oct 30 21:37:07.510315 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:07.510286 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" Oct 30 21:37:09.333205 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:09.333171 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf"] Oct 30 21:37:09.336298 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:09.336272 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" Oct 30 21:37:09.386003 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:09.385976 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-554f46\"/\"premium-w4-554f46-kserve-self-signed-certs\"" Oct 30 21:37:09.433938 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:09.433907 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf"] Oct 30 21:37:09.441445 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:09.441404 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-model-cache\") pod \"premium-w4-554f46-kserve-6848cd5dfb-8j9vf\" (UID: \"0155a02e-8df7-4847-b40f-b8d9e4bac8c6\") " pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" Oct 30 21:37:09.441571 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:09.441467 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-tmp-dir\") pod \"premium-w4-554f46-kserve-6848cd5dfb-8j9vf\" (UID: \"0155a02e-8df7-4847-b40f-b8d9e4bac8c6\") " pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" Oct 30 21:37:09.441571 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:09.441550 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qclqz\" (UniqueName: \"kubernetes.io/projected/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-kube-api-access-qclqz\") pod \"premium-w4-554f46-kserve-6848cd5dfb-8j9vf\" (UID: \"0155a02e-8df7-4847-b40f-b8d9e4bac8c6\") " pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" Oct 30 21:37:09.441652 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:09.441587 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-tls-certs\") pod \"premium-w4-554f46-kserve-6848cd5dfb-8j9vf\" (UID: \"0155a02e-8df7-4847-b40f-b8d9e4bac8c6\") " pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" Oct 30 21:37:09.441652 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:09.441625 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-home\") pod \"premium-w4-554f46-kserve-6848cd5dfb-8j9vf\" (UID: \"0155a02e-8df7-4847-b40f-b8d9e4bac8c6\") " pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" Oct 30 21:37:09.441742 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:09.441732 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-dshm\") pod \"premium-w4-554f46-kserve-6848cd5dfb-8j9vf\" (UID: \"0155a02e-8df7-4847-b40f-b8d9e4bac8c6\") " pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" Oct 30 21:37:09.542423 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:09.542374 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-model-cache\") pod \"premium-w4-554f46-kserve-6848cd5dfb-8j9vf\" (UID: \"0155a02e-8df7-4847-b40f-b8d9e4bac8c6\") " pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" Oct 30 21:37:09.542423 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:09.542431 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-tmp-dir\") pod \"premium-w4-554f46-kserve-6848cd5dfb-8j9vf\" (UID: \"0155a02e-8df7-4847-b40f-b8d9e4bac8c6\") " pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" Oct 30 21:37:09.542683 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:09.542463 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qclqz\" (UniqueName: \"kubernetes.io/projected/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-kube-api-access-qclqz\") pod \"premium-w4-554f46-kserve-6848cd5dfb-8j9vf\" (UID: \"0155a02e-8df7-4847-b40f-b8d9e4bac8c6\") " pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" Oct 30 21:37:09.542683 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:09.542483 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-tls-certs\") pod \"premium-w4-554f46-kserve-6848cd5dfb-8j9vf\" (UID: \"0155a02e-8df7-4847-b40f-b8d9e4bac8c6\") " pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" Oct 30 21:37:09.542683 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:09.542532 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-home\") pod \"premium-w4-554f46-kserve-6848cd5dfb-8j9vf\" (UID: \"0155a02e-8df7-4847-b40f-b8d9e4bac8c6\") " pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" Oct 30 21:37:09.542683 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:09.542592 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-dshm\") pod \"premium-w4-554f46-kserve-6848cd5dfb-8j9vf\" (UID: \"0155a02e-8df7-4847-b40f-b8d9e4bac8c6\") " pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" Oct 30 21:37:09.542896 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:09.542866 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-model-cache\") pod \"premium-w4-554f46-kserve-6848cd5dfb-8j9vf\" (UID: \"0155a02e-8df7-4847-b40f-b8d9e4bac8c6\") " pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" Oct 30 21:37:09.542966 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:09.542896 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-tmp-dir\") pod \"premium-w4-554f46-kserve-6848cd5dfb-8j9vf\" (UID: \"0155a02e-8df7-4847-b40f-b8d9e4bac8c6\") " pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" Oct 30 21:37:09.543008 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:09.542984 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-home\") pod \"premium-w4-554f46-kserve-6848cd5dfb-8j9vf\" (UID: \"0155a02e-8df7-4847-b40f-b8d9e4bac8c6\") " pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" Oct 30 21:37:09.544855 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:09.544830 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-dshm\") pod \"premium-w4-554f46-kserve-6848cd5dfb-8j9vf\" (UID: \"0155a02e-8df7-4847-b40f-b8d9e4bac8c6\") " pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" Oct 30 21:37:09.545831 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:09.545808 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-tls-certs\") pod \"premium-w4-554f46-kserve-6848cd5dfb-8j9vf\" (UID: \"0155a02e-8df7-4847-b40f-b8d9e4bac8c6\") " pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" Oct 30 21:37:09.603327 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:09.603244 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qclqz\" (UniqueName: \"kubernetes.io/projected/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-kube-api-access-qclqz\") pod \"premium-w4-554f46-kserve-6848cd5dfb-8j9vf\" (UID: \"0155a02e-8df7-4847-b40f-b8d9e4bac8c6\") " pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" Oct 30 21:37:09.648616 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:09.648575 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" Oct 30 21:37:10.039732 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:10.039695 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf"] Oct 30 21:37:10.046416 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:37:10.046390 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0155a02e_8df7_4847_b40f_b8d9e4bac8c6.slice/crio-4707bdff7c862e4b927c08e73236972d6cbbe7636003dbcecd8ecfb9750b0d26 WatchSource:0}: Error finding container 4707bdff7c862e4b927c08e73236972d6cbbe7636003dbcecd8ecfb9750b0d26: Status 404 returned error can't find the container with id 4707bdff7c862e4b927c08e73236972d6cbbe7636003dbcecd8ecfb9750b0d26 Oct 30 21:37:10.513095 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:10.513058 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" event={"ID":"0155a02e-8df7-4847-b40f-b8d9e4bac8c6","Type":"ContainerStarted","Data":"7cb8434476e1f9021cba0ba7392104f1bf5032786bce784254ed2755dfc5c9f4"} Oct 30 21:37:10.513095 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:10.513102 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" event={"ID":"0155a02e-8df7-4847-b40f-b8d9e4bac8c6","Type":"ContainerStarted","Data":"4707bdff7c862e4b927c08e73236972d6cbbe7636003dbcecd8ecfb9750b0d26"} Oct 30 21:37:10.625633 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:10.625580 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" podStartSLOduration=1.625565363 podStartE2EDuration="1.625565363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:10.624610394 +0000 UTC m=+1406.179310170" watchObservedRunningTime="2026-10-30 21:37:10.625565363 +0000 UTC m=+1406.180265152" Oct 30 21:37:10.648827 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:10.648794 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" Oct 30 21:37:12.638207 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:12.638174 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87"] Oct 30 21:37:12.641663 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:12.641631 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" Oct 30 21:37:12.671687 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:12.671656 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qbtf\" (UniqueName: \"kubernetes.io/projected/4dee11ef-9647-47ee-a4b1-686680256542-kube-api-access-6qbtf\") pod \"simulator-w2-ed76b8-kserve-6f96c5c447-6qk87\" (UID: \"4dee11ef-9647-47ee-a4b1-686680256542\") " pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" Oct 30 21:37:12.671854 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:12.671693 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dee11ef-9647-47ee-a4b1-686680256542-home\") pod \"simulator-w2-ed76b8-kserve-6f96c5c447-6qk87\" (UID: \"4dee11ef-9647-47ee-a4b1-686680256542\") " pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" Oct 30 21:37:12.671854 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:12.671826 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dee11ef-9647-47ee-a4b1-686680256542-tls-certs\") pod \"simulator-w2-ed76b8-kserve-6f96c5c447-6qk87\" (UID: \"4dee11ef-9647-47ee-a4b1-686680256542\") " pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" Oct 30 21:37:12.671972 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:12.671855 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dee11ef-9647-47ee-a4b1-686680256542-tmp-dir\") pod \"simulator-w2-ed76b8-kserve-6f96c5c447-6qk87\" (UID: \"4dee11ef-9647-47ee-a4b1-686680256542\") " pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" Oct 30 21:37:12.671972 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:12.671875 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dee11ef-9647-47ee-a4b1-686680256542-model-cache\") pod \"simulator-w2-ed76b8-kserve-6f96c5c447-6qk87\" (UID: \"4dee11ef-9647-47ee-a4b1-686680256542\") " pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" Oct 30 21:37:12.672081 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:12.672013 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dee11ef-9647-47ee-a4b1-686680256542-dshm\") pod \"simulator-w2-ed76b8-kserve-6f96c5c447-6qk87\" (UID: \"4dee11ef-9647-47ee-a4b1-686680256542\") " pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" Oct 30 21:37:12.737267 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:12.737233 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-ed76b8\"/\"kube-root-ca.crt\"" Oct 30 21:37:12.737490 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:12.737440 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ed76b8\"/\"simulator-w2-ed76b8-kserve-self-signed-certs\"" Oct 30 21:37:12.737490 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:12.737474 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-ed76b8\"/\"openshift-service-ca.crt\"" Oct 30 21:37:12.737591 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:12.737478 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ed76b8\"/\"default-dockercfg-wpc6v\"" Oct 30 21:37:12.739555 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:12.739535 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87"] Oct 30 21:37:12.773514 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:12.773476 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dee11ef-9647-47ee-a4b1-686680256542-tmp-dir\") pod \"simulator-w2-ed76b8-kserve-6f96c5c447-6qk87\" (UID: \"4dee11ef-9647-47ee-a4b1-686680256542\") " pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" Oct 30 21:37:12.773514 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:12.773516 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dee11ef-9647-47ee-a4b1-686680256542-model-cache\") pod \"simulator-w2-ed76b8-kserve-6f96c5c447-6qk87\" (UID: \"4dee11ef-9647-47ee-a4b1-686680256542\") " pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" Oct 30 21:37:12.773803 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:12.773560 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dee11ef-9647-47ee-a4b1-686680256542-dshm\") pod \"simulator-w2-ed76b8-kserve-6f96c5c447-6qk87\" (UID: \"4dee11ef-9647-47ee-a4b1-686680256542\") " pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" Oct 30 21:37:12.773803 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:12.773599 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qbtf\" (UniqueName: \"kubernetes.io/projected/4dee11ef-9647-47ee-a4b1-686680256542-kube-api-access-6qbtf\") pod \"simulator-w2-ed76b8-kserve-6f96c5c447-6qk87\" (UID: \"4dee11ef-9647-47ee-a4b1-686680256542\") " pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" Oct 30 21:37:12.773803 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:12.773629 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dee11ef-9647-47ee-a4b1-686680256542-home\") pod \"simulator-w2-ed76b8-kserve-6f96c5c447-6qk87\" (UID: \"4dee11ef-9647-47ee-a4b1-686680256542\") " pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" Oct 30 21:37:12.773803 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:12.773725 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dee11ef-9647-47ee-a4b1-686680256542-tls-certs\") pod \"simulator-w2-ed76b8-kserve-6f96c5c447-6qk87\" (UID: \"4dee11ef-9647-47ee-a4b1-686680256542\") " pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" Oct 30 21:37:12.775039 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:12.774011 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dee11ef-9647-47ee-a4b1-686680256542-tmp-dir\") pod \"simulator-w2-ed76b8-kserve-6f96c5c447-6qk87\" (UID: \"4dee11ef-9647-47ee-a4b1-686680256542\") " pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" Oct 30 21:37:12.775039 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:12.774048 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dee11ef-9647-47ee-a4b1-686680256542-model-cache\") pod \"simulator-w2-ed76b8-kserve-6f96c5c447-6qk87\" (UID: \"4dee11ef-9647-47ee-a4b1-686680256542\") " pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" Oct 30 21:37:12.775039 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:12.774227 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dee11ef-9647-47ee-a4b1-686680256542-home\") pod \"simulator-w2-ed76b8-kserve-6f96c5c447-6qk87\" (UID: \"4dee11ef-9647-47ee-a4b1-686680256542\") " pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" Oct 30 21:37:12.776594 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:12.776568 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dee11ef-9647-47ee-a4b1-686680256542-dshm\") pod \"simulator-w2-ed76b8-kserve-6f96c5c447-6qk87\" (UID: \"4dee11ef-9647-47ee-a4b1-686680256542\") " pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" Oct 30 21:37:12.776966 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:12.776947 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dee11ef-9647-47ee-a4b1-686680256542-tls-certs\") pod \"simulator-w2-ed76b8-kserve-6f96c5c447-6qk87\" (UID: \"4dee11ef-9647-47ee-a4b1-686680256542\") " pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" Oct 30 21:37:12.838813 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:12.838776 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qbtf\" (UniqueName: \"kubernetes.io/projected/4dee11ef-9647-47ee-a4b1-686680256542-kube-api-access-6qbtf\") pod \"simulator-w2-ed76b8-kserve-6f96c5c447-6qk87\" (UID: \"4dee11ef-9647-47ee-a4b1-686680256542\") " pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" Oct 30 21:37:12.955943 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:12.955853 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" Oct 30 21:37:13.361043 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:13.361006 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87"] Oct 30 21:37:13.367524 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:37:13.367495 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4dee11ef_9647_47ee_a4b1_686680256542.slice/crio-9d15a58f3e1b775a47fb459d58872ca4d74f9178804f8e6d60cf50418c4a1503 WatchSource:0}: Error finding container 9d15a58f3e1b775a47fb459d58872ca4d74f9178804f8e6d60cf50418c4a1503: Status 404 returned error can't find the container with id 9d15a58f3e1b775a47fb459d58872ca4d74f9178804f8e6d60cf50418c4a1503 Oct 30 21:37:13.527114 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:13.527079 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" event={"ID":"4dee11ef-9647-47ee-a4b1-686680256542","Type":"ContainerStarted","Data":"9b78d5b264887a2ae01f70c4ffe21587c242e470eead7b603c2191bdf76e4c55"} Oct 30 21:37:13.527310 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:13.527124 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" event={"ID":"4dee11ef-9647-47ee-a4b1-686680256542","Type":"ContainerStarted","Data":"9d15a58f3e1b775a47fb459d58872ca4d74f9178804f8e6d60cf50418c4a1503"} Oct 30 21:37:13.641371 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:13.641220 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" podStartSLOduration=1.641199627 podStartE2EDuration="1.641199627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:13.637977462 +0000 UTC m=+1409.192677242" watchObservedRunningTime="2026-10-30 21:37:13.641199627 +0000 UTC m=+1409.195899396" Oct 30 21:37:13.956885 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:13.956800 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" Oct 30 21:37:19.649540 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:19.649508 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" Oct 30 21:37:19.662402 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:19.662377 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" Oct 30 21:37:19.683827 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:37:19.683783 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:37:20.565844 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:20.565812 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" Oct 30 21:37:22.956112 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:22.956076 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" Oct 30 21:37:22.968721 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:22.968700 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" Oct 30 21:37:23.578237 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:23.578207 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" Oct 30 21:37:26.322426 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:26.322394 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:37:38.875739 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:38.875692 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx"] Oct 30 21:37:38.879170 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:38.879146 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" Oct 30 21:37:38.929078 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:38.929048 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-afbe5b\"/\"kube-root-ca.crt\"" Oct 30 21:37:38.950022 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:38.949983 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-afbe5b\"/\"openshift-service-ca.crt\"" Oct 30 21:37:38.975870 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:38.975827 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-afbe5b\"/\"default-dockercfg-f277b\"" Oct 30 21:37:38.976057 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:38.975911 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-afbe5b\"/\"simulator-w0-afbe5b-kserve-self-signed-certs\"" Oct 30 21:37:38.977349 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:38.977317 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx"] Oct 30 21:37:39.005121 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:39.005078 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90bfb1e4-9526-40f6-aed4-ee46c2d86471-model-cache\") pod \"simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx\" (UID: \"90bfb1e4-9526-40f6-aed4-ee46c2d86471\") " pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" Oct 30 21:37:39.005278 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:39.005144 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90bfb1e4-9526-40f6-aed4-ee46c2d86471-tmp-dir\") pod \"simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx\" (UID: \"90bfb1e4-9526-40f6-aed4-ee46c2d86471\") " pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" Oct 30 21:37:39.005278 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:39.005246 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90bfb1e4-9526-40f6-aed4-ee46c2d86471-dshm\") pod \"simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx\" (UID: \"90bfb1e4-9526-40f6-aed4-ee46c2d86471\") " pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" Oct 30 21:37:39.005278 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:39.005269 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90bfb1e4-9526-40f6-aed4-ee46c2d86471-tls-certs\") pod \"simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx\" (UID: \"90bfb1e4-9526-40f6-aed4-ee46c2d86471\") " pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" Oct 30 21:37:39.005437 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:39.005294 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9hd6\" (UniqueName: \"kubernetes.io/projected/90bfb1e4-9526-40f6-aed4-ee46c2d86471-kube-api-access-p9hd6\") pod \"simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx\" (UID: \"90bfb1e4-9526-40f6-aed4-ee46c2d86471\") " pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" Oct 30 21:37:39.005437 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:39.005385 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90bfb1e4-9526-40f6-aed4-ee46c2d86471-home\") pod \"simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx\" (UID: \"90bfb1e4-9526-40f6-aed4-ee46c2d86471\") " pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" Oct 30 21:37:39.106690 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:39.106648 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90bfb1e4-9526-40f6-aed4-ee46c2d86471-dshm\") pod \"simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx\" (UID: \"90bfb1e4-9526-40f6-aed4-ee46c2d86471\") " pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" Oct 30 21:37:39.106690 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:39.106695 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90bfb1e4-9526-40f6-aed4-ee46c2d86471-tls-certs\") pod \"simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx\" (UID: \"90bfb1e4-9526-40f6-aed4-ee46c2d86471\") " pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" Oct 30 21:37:39.106961 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:39.106715 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9hd6\" (UniqueName: \"kubernetes.io/projected/90bfb1e4-9526-40f6-aed4-ee46c2d86471-kube-api-access-p9hd6\") pod \"simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx\" (UID: \"90bfb1e4-9526-40f6-aed4-ee46c2d86471\") " pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" Oct 30 21:37:39.106961 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:39.106850 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90bfb1e4-9526-40f6-aed4-ee46c2d86471-home\") pod \"simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx\" (UID: \"90bfb1e4-9526-40f6-aed4-ee46c2d86471\") " pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" Oct 30 21:37:39.106961 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:39.106916 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90bfb1e4-9526-40f6-aed4-ee46c2d86471-model-cache\") pod \"simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx\" (UID: \"90bfb1e4-9526-40f6-aed4-ee46c2d86471\") " pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" Oct 30 21:37:39.107118 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:39.106968 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90bfb1e4-9526-40f6-aed4-ee46c2d86471-tmp-dir\") pod \"simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx\" (UID: \"90bfb1e4-9526-40f6-aed4-ee46c2d86471\") " pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" Oct 30 21:37:39.107401 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:39.107376 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90bfb1e4-9526-40f6-aed4-ee46c2d86471-model-cache\") pod \"simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx\" (UID: \"90bfb1e4-9526-40f6-aed4-ee46c2d86471\") " pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" Oct 30 21:37:39.107492 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:39.107422 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90bfb1e4-9526-40f6-aed4-ee46c2d86471-tmp-dir\") pod \"simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx\" (UID: \"90bfb1e4-9526-40f6-aed4-ee46c2d86471\") " pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" Oct 30 21:37:39.107644 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:39.107625 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90bfb1e4-9526-40f6-aed4-ee46c2d86471-home\") pod \"simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx\" (UID: \"90bfb1e4-9526-40f6-aed4-ee46c2d86471\") " pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" Oct 30 21:37:39.109236 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:39.109209 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90bfb1e4-9526-40f6-aed4-ee46c2d86471-dshm\") pod \"simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx\" (UID: \"90bfb1e4-9526-40f6-aed4-ee46c2d86471\") " pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" Oct 30 21:37:39.109502 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:39.109483 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90bfb1e4-9526-40f6-aed4-ee46c2d86471-tls-certs\") pod \"simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx\" (UID: \"90bfb1e4-9526-40f6-aed4-ee46c2d86471\") " pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" Oct 30 21:37:39.167227 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:39.167143 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9hd6\" (UniqueName: \"kubernetes.io/projected/90bfb1e4-9526-40f6-aed4-ee46c2d86471-kube-api-access-p9hd6\") pod \"simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx\" (UID: \"90bfb1e4-9526-40f6-aed4-ee46c2d86471\") " pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" Oct 30 21:37:39.190888 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:39.190845 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" Oct 30 21:37:39.568624 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:39.568591 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx"] Oct 30 21:37:39.575834 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:37:39.575808 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90bfb1e4_9526_40f6_aed4_ee46c2d86471.slice/crio-9d6436214a40f8f2fde67686c4fa6861e700314abb6e0ba4575c43d8f5fa3830 WatchSource:0}: Error finding container 9d6436214a40f8f2fde67686c4fa6861e700314abb6e0ba4575c43d8f5fa3830: Status 404 returned error can't find the container with id 9d6436214a40f8f2fde67686c4fa6861e700314abb6e0ba4575c43d8f5fa3830 Oct 30 21:37:39.635368 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:39.635317 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" event={"ID":"90bfb1e4-9526-40f6-aed4-ee46c2d86471","Type":"ContainerStarted","Data":"83471276efe32bda9d7935fc7b28bf80bc1c1a901efd72e572720261129c37bb"} Oct 30 21:37:39.635491 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:39.635385 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" event={"ID":"90bfb1e4-9526-40f6-aed4-ee46c2d86471","Type":"ContainerStarted","Data":"9d6436214a40f8f2fde67686c4fa6861e700314abb6e0ba4575c43d8f5fa3830"} Oct 30 21:37:39.752546 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:39.752492 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" podStartSLOduration=1.752478815 podStartE2EDuration="1.752478815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:39.751165602 +0000 UTC m=+1435.305865379" watchObservedRunningTime="2026-10-30 21:37:39.752478815 +0000 UTC m=+1435.307178591" Oct 30 21:37:40.191281 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:40.191251 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" Oct 30 21:37:44.580972 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:44.580942 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p"] Oct 30 21:37:44.585220 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:44.585199 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" Oct 30 21:37:44.680702 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:44.680670 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a2dced\"/\"premium-w6-a2dced-kserve-self-signed-certs\"" Oct 30 21:37:44.682176 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:44.682156 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p"] Oct 30 21:37:44.760537 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:44.760509 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-tls-certs\") pod \"premium-w6-a2dced-kserve-86778c4b8f-jbn5p\" (UID: \"8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3\") " pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" Oct 30 21:37:44.760693 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:44.760556 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-dshm\") pod \"premium-w6-a2dced-kserve-86778c4b8f-jbn5p\" (UID: \"8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3\") " pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" Oct 30 21:37:44.760693 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:44.760646 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f64c\" (UniqueName: \"kubernetes.io/projected/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-kube-api-access-7f64c\") pod \"premium-w6-a2dced-kserve-86778c4b8f-jbn5p\" (UID: \"8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3\") " pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" Oct 30 21:37:44.760765 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:44.760697 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-home\") pod \"premium-w6-a2dced-kserve-86778c4b8f-jbn5p\" (UID: \"8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3\") " pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" Oct 30 21:37:44.760765 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:44.760724 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-tmp-dir\") pod \"premium-w6-a2dced-kserve-86778c4b8f-jbn5p\" (UID: \"8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3\") " pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" Oct 30 21:37:44.760765 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:44.760760 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-model-cache\") pod \"premium-w6-a2dced-kserve-86778c4b8f-jbn5p\" (UID: \"8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3\") " pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" Oct 30 21:37:44.861964 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:44.861864 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-home\") pod \"premium-w6-a2dced-kserve-86778c4b8f-jbn5p\" (UID: \"8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3\") " pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" Oct 30 21:37:44.861964 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:44.861922 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-tmp-dir\") pod \"premium-w6-a2dced-kserve-86778c4b8f-jbn5p\" (UID: \"8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3\") " pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" Oct 30 21:37:44.862226 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:44.861977 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-model-cache\") pod \"premium-w6-a2dced-kserve-86778c4b8f-jbn5p\" (UID: \"8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3\") " pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" Oct 30 21:37:44.862226 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:44.862020 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-tls-certs\") pod \"premium-w6-a2dced-kserve-86778c4b8f-jbn5p\" (UID: \"8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3\") " pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" Oct 30 21:37:44.862226 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:44.862070 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-dshm\") pod \"premium-w6-a2dced-kserve-86778c4b8f-jbn5p\" (UID: \"8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3\") " pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" Oct 30 21:37:44.862226 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:44.862122 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f64c\" (UniqueName: \"kubernetes.io/projected/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-kube-api-access-7f64c\") pod \"premium-w6-a2dced-kserve-86778c4b8f-jbn5p\" (UID: \"8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3\") " pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" Oct 30 21:37:44.862474 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:44.862446 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-home\") pod \"premium-w6-a2dced-kserve-86778c4b8f-jbn5p\" (UID: \"8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3\") " pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" Oct 30 21:37:44.862538 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:44.862463 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-tmp-dir\") pod \"premium-w6-a2dced-kserve-86778c4b8f-jbn5p\" (UID: \"8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3\") " pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" Oct 30 21:37:44.862739 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:44.862716 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-model-cache\") pod \"premium-w6-a2dced-kserve-86778c4b8f-jbn5p\" (UID: \"8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3\") " pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" Oct 30 21:37:44.865169 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:44.865140 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-dshm\") pod \"premium-w6-a2dced-kserve-86778c4b8f-jbn5p\" (UID: \"8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3\") " pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" Oct 30 21:37:44.865454 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:44.865433 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-tls-certs\") pod \"premium-w6-a2dced-kserve-86778c4b8f-jbn5p\" (UID: \"8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3\") " pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" Oct 30 21:37:44.920601 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:44.920565 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f64c\" (UniqueName: \"kubernetes.io/projected/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-kube-api-access-7f64c\") pod \"premium-w6-a2dced-kserve-86778c4b8f-jbn5p\" (UID: \"8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3\") " pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" Oct 30 21:37:45.197082 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:45.196998 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" Oct 30 21:37:45.566240 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:45.566201 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p"] Oct 30 21:37:45.574860 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:37:45.574835 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8cc70a2a_65b6_4dbc_b6e1_1bedf8354fe3.slice/crio-e26ec883712965a4d231cfb4fbcf41e5baaef25bec6b604309ec4aad3de0b899 WatchSource:0}: Error finding container e26ec883712965a4d231cfb4fbcf41e5baaef25bec6b604309ec4aad3de0b899: Status 404 returned error can't find the container with id e26ec883712965a4d231cfb4fbcf41e5baaef25bec6b604309ec4aad3de0b899 Oct 30 21:37:45.662003 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:45.661962 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" event={"ID":"8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3","Type":"ContainerStarted","Data":"720a70d747347c95ce3b6c6f00aa7fd196bcde1c7d7664f352102da28ea3bc79"} Oct 30 21:37:45.662003 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:45.662002 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" event={"ID":"8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3","Type":"ContainerStarted","Data":"e26ec883712965a4d231cfb4fbcf41e5baaef25bec6b604309ec4aad3de0b899"} Oct 30 21:37:45.774088 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:45.774033 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" podStartSLOduration=1.7740188140000002 podStartE2EDuration="1.774018814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:45.772856871 +0000 UTC m=+1441.327556648" watchObservedRunningTime="2026-10-30 21:37:45.774018814 +0000 UTC m=+1441.328718591" Oct 30 21:37:46.197106 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:46.197065 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" Oct 30 21:37:49.191516 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:49.191480 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" Oct 30 21:37:49.203885 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:49.203861 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" Oct 30 21:37:49.464745 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:49.464667 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:37:49.685224 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:37:49.685187 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:37:49.690843 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:49.690824 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" Oct 30 21:37:55.197672 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:55.197637 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" Oct 30 21:37:55.210302 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:55.210272 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" Oct 30 21:37:55.714754 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:37:55.714720 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" Oct 30 21:38:19.684794 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:38:19.684757 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:38:21.964700 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:21.964668 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:38:39.298205 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.298170 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc"] Oct 30 21:38:39.301414 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.301387 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" Oct 30 21:38:39.370058 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.370023 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-tmp-dir\") pod \"premium-w2-ed76b8-kserve-5586cc459d-4lnkc\" (UID: \"e3281dbe-e8f9-4bab-9ae4-2279e79b7292\") " pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" Oct 30 21:38:39.370215 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.370103 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-dshm\") pod \"premium-w2-ed76b8-kserve-5586cc459d-4lnkc\" (UID: \"e3281dbe-e8f9-4bab-9ae4-2279e79b7292\") " pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" Oct 30 21:38:39.370215 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.370131 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-model-cache\") pod \"premium-w2-ed76b8-kserve-5586cc459d-4lnkc\" (UID: \"e3281dbe-e8f9-4bab-9ae4-2279e79b7292\") " pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" Oct 30 21:38:39.370328 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.370251 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfksb\" (UniqueName: \"kubernetes.io/projected/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-kube-api-access-rfksb\") pod \"premium-w2-ed76b8-kserve-5586cc459d-4lnkc\" (UID: \"e3281dbe-e8f9-4bab-9ae4-2279e79b7292\") " pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" Oct 30 21:38:39.370328 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.370320 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-tls-certs\") pod \"premium-w2-ed76b8-kserve-5586cc459d-4lnkc\" (UID: \"e3281dbe-e8f9-4bab-9ae4-2279e79b7292\") " pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" Oct 30 21:38:39.370442 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.370419 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-home\") pod \"premium-w2-ed76b8-kserve-5586cc459d-4lnkc\" (UID: \"e3281dbe-e8f9-4bab-9ae4-2279e79b7292\") " pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" Oct 30 21:38:39.396864 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.396837 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ed76b8\"/\"premium-w2-ed76b8-kserve-self-signed-certs\"" Oct 30 21:38:39.398422 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.398395 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc"] Oct 30 21:38:39.445532 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.445509 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7378b486-6p5r9"] Oct 30 21:38:39.455595 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.455571 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7378b486-6p5r9" Oct 30 21:38:39.470986 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.470958 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-dshm\") pod \"premium-w2-ed76b8-kserve-5586cc459d-4lnkc\" (UID: \"e3281dbe-e8f9-4bab-9ae4-2279e79b7292\") " pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" Oct 30 21:38:39.471113 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.470994 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-model-cache\") pod \"premium-w2-ed76b8-kserve-5586cc459d-4lnkc\" (UID: \"e3281dbe-e8f9-4bab-9ae4-2279e79b7292\") " pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" Oct 30 21:38:39.471113 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.471103 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfksb\" (UniqueName: \"kubernetes.io/projected/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-kube-api-access-rfksb\") pod \"premium-w2-ed76b8-kserve-5586cc459d-4lnkc\" (UID: \"e3281dbe-e8f9-4bab-9ae4-2279e79b7292\") " pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" Oct 30 21:38:39.471189 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.471159 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-tls-certs\") pod \"premium-w2-ed76b8-kserve-5586cc459d-4lnkc\" (UID: \"e3281dbe-e8f9-4bab-9ae4-2279e79b7292\") " pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" Oct 30 21:38:39.471223 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.471189 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfhwr\" (UniqueName: \"kubernetes.io/projected/dc5a7f12-8474-49fa-b5ba-b3db497c0483-kube-api-access-tfhwr\") pod \"maas-api-revoke-sub-7378b486-6p5r9\" (UID: \"dc5a7f12-8474-49fa-b5ba-b3db497c0483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7378b486-6p5r9" Oct 30 21:38:39.471263 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.471220 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc5a7f12-8474-49fa-b5ba-b3db497c0483-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7378b486-6p5r9\" (UID: \"dc5a7f12-8474-49fa-b5ba-b3db497c0483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7378b486-6p5r9" Oct 30 21:38:39.471263 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.471246 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc5a7f12-8474-49fa-b5ba-b3db497c0483-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7378b486-6p5r9\" (UID: \"dc5a7f12-8474-49fa-b5ba-b3db497c0483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7378b486-6p5r9" Oct 30 21:38:39.471386 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.471302 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-home\") pod \"premium-w2-ed76b8-kserve-5586cc459d-4lnkc\" (UID: \"e3281dbe-e8f9-4bab-9ae4-2279e79b7292\") " pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" Oct 30 21:38:39.471450 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.471385 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-tmp-dir\") pod \"premium-w2-ed76b8-kserve-5586cc459d-4lnkc\" (UID: \"e3281dbe-e8f9-4bab-9ae4-2279e79b7292\") " pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" Oct 30 21:38:39.471450 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.471420 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-model-cache\") pod \"premium-w2-ed76b8-kserve-5586cc459d-4lnkc\" (UID: \"e3281dbe-e8f9-4bab-9ae4-2279e79b7292\") " pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" Oct 30 21:38:39.471589 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.471567 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-home\") pod \"premium-w2-ed76b8-kserve-5586cc459d-4lnkc\" (UID: \"e3281dbe-e8f9-4bab-9ae4-2279e79b7292\") " pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" Oct 30 21:38:39.471674 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.471659 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-tmp-dir\") pod \"premium-w2-ed76b8-kserve-5586cc459d-4lnkc\" (UID: \"e3281dbe-e8f9-4bab-9ae4-2279e79b7292\") " pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" Oct 30 21:38:39.473177 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.473151 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-dshm\") pod \"premium-w2-ed76b8-kserve-5586cc459d-4lnkc\" (UID: \"e3281dbe-e8f9-4bab-9ae4-2279e79b7292\") " pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" Oct 30 21:38:39.473628 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.473610 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-tls-certs\") pod \"premium-w2-ed76b8-kserve-5586cc459d-4lnkc\" (UID: \"e3281dbe-e8f9-4bab-9ae4-2279e79b7292\") " pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" Oct 30 21:38:39.492511 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.492492 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7378b486-6p5r9"] Oct 30 21:38:39.505534 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.505481 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:38:39.505646 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.505532 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:38:39.505646 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.505595 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:38:39.505646 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.505601 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:38:39.529044 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.528985 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfksb\" (UniqueName: \"kubernetes.io/projected/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-kube-api-access-rfksb\") pod \"premium-w2-ed76b8-kserve-5586cc459d-4lnkc\" (UID: \"e3281dbe-e8f9-4bab-9ae4-2279e79b7292\") " pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" Oct 30 21:38:39.572648 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.572612 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfhwr\" (UniqueName: \"kubernetes.io/projected/dc5a7f12-8474-49fa-b5ba-b3db497c0483-kube-api-access-tfhwr\") pod \"maas-api-revoke-sub-7378b486-6p5r9\" (UID: \"dc5a7f12-8474-49fa-b5ba-b3db497c0483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7378b486-6p5r9" Oct 30 21:38:39.572808 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.572662 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc5a7f12-8474-49fa-b5ba-b3db497c0483-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7378b486-6p5r9\" (UID: \"dc5a7f12-8474-49fa-b5ba-b3db497c0483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7378b486-6p5r9" Oct 30 21:38:39.572808 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.572703 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc5a7f12-8474-49fa-b5ba-b3db497c0483-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7378b486-6p5r9\" (UID: \"dc5a7f12-8474-49fa-b5ba-b3db497c0483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7378b486-6p5r9" Oct 30 21:38:39.573315 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.573288 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc5a7f12-8474-49fa-b5ba-b3db497c0483-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7378b486-6p5r9\" (UID: \"dc5a7f12-8474-49fa-b5ba-b3db497c0483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7378b486-6p5r9" Oct 30 21:38:39.575262 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.575235 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc5a7f12-8474-49fa-b5ba-b3db497c0483-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7378b486-6p5r9\" (UID: \"dc5a7f12-8474-49fa-b5ba-b3db497c0483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7378b486-6p5r9" Oct 30 21:38:39.614152 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.614125 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" Oct 30 21:38:39.631043 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.631016 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfhwr\" (UniqueName: \"kubernetes.io/projected/dc5a7f12-8474-49fa-b5ba-b3db497c0483-kube-api-access-tfhwr\") pod \"maas-api-revoke-sub-7378b486-6p5r9\" (UID: \"dc5a7f12-8474-49fa-b5ba-b3db497c0483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7378b486-6p5r9" Oct 30 21:38:39.772975 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.772944 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7378b486-6p5r9" Oct 30 21:38:39.993353 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:39.993309 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc"] Oct 30 21:38:39.999804 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:38:39.999781 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3281dbe_e8f9_4bab_9ae4_2279e79b7292.slice/crio-ec63e28ce9b2a8683a6fd9cf5581f37d6958d986f697abecacca629297dc354d WatchSource:0}: Error finding container ec63e28ce9b2a8683a6fd9cf5581f37d6958d986f697abecacca629297dc354d: Status 404 returned error can't find the container with id ec63e28ce9b2a8683a6fd9cf5581f37d6958d986f697abecacca629297dc354d Oct 30 21:38:40.139562 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:40.139533 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7378b486-6p5r9"] Oct 30 21:38:40.146646 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:38:40.146621 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc5a7f12_8474_49fa_b5ba_b3db497c0483.slice/crio-352f878fee2318e0f6c0d5de05c76a598ca15a8c33ced15cd36a83217e857fc8 WatchSource:0}: Error finding container 352f878fee2318e0f6c0d5de05c76a598ca15a8c33ced15cd36a83217e857fc8: Status 404 returned error can't find the container with id 352f878fee2318e0f6c0d5de05c76a598ca15a8c33ced15cd36a83217e857fc8 Oct 30 21:38:40.906307 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:40.906273 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7378b486-6p5r9" event={"ID":"dc5a7f12-8474-49fa-b5ba-b3db497c0483","Type":"ContainerStarted","Data":"352f878fee2318e0f6c0d5de05c76a598ca15a8c33ced15cd36a83217e857fc8"} Oct 30 21:38:40.907951 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:40.907925 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" event={"ID":"e3281dbe-e8f9-4bab-9ae4-2279e79b7292","Type":"ContainerStarted","Data":"44bb8b61144f12209f730c543869497701cc5a3c15dadb47a4a83a781912a01e"} Oct 30 21:38:40.908084 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:40.907962 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" event={"ID":"e3281dbe-e8f9-4bab-9ae4-2279e79b7292","Type":"ContainerStarted","Data":"ec63e28ce9b2a8683a6fd9cf5581f37d6958d986f697abecacca629297dc354d"} Oct 30 21:38:41.020482 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:41.020431 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" podStartSLOduration=2.020417344 podStartE2EDuration="2.020417344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:41.018489634 +0000 UTC m=+1496.573189412" watchObservedRunningTime="2026-10-30 21:38:41.020417344 +0000 UTC m=+1496.575117183" Oct 30 21:38:41.295422 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:41.295389 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:38:41.615134 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:41.615052 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" Oct 30 21:38:44.931421 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:44.931378 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7378b486-6p5r9" event={"ID":"dc5a7f12-8474-49fa-b5ba-b3db497c0483","Type":"ContainerStarted","Data":"f2b8f0f6398ee28447a619aa3b7e983b36fc263f4f857a611f31c8e605718317"} Oct 30 21:38:45.041810 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:45.041758 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7378b486-6p5r9" podStartSLOduration=2.085982017 podStartE2EDuration="6.04174128s" totalImagesPullingTime="3.955759263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:45.041055467 +0000 UTC m=+1500.595755245" watchObservedRunningTime="2026-10-30 21:38:45.04174128 +0000 UTC m=+1500.596441058" Oct 30 21:38:45.937220 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:45.937191 4560 generic.go:356] "Generic (PLEG): container finished" podID="dc5a7f12-8474-49fa-b5ba-b3db497c0483" containerID="f2b8f0f6398ee28447a619aa3b7e983b36fc263f4f857a611f31c8e605718317" exitCode=0 Oct 30 21:38:45.937618 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:45.937281 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7378b486-6p5r9" event={"ID":"dc5a7f12-8474-49fa-b5ba-b3db497c0483","Type":"ContainerDied","Data":"f2b8f0f6398ee28447a619aa3b7e983b36fc263f4f857a611f31c8e605718317"} Oct 30 21:38:47.071714 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:47.071691 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7378b486-6p5r9" Oct 30 21:38:47.145140 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:47.145109 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc5a7f12-8474-49fa-b5ba-b3db497c0483-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc5a7f12-8474-49fa-b5ba-b3db497c0483-maas-api-xks-ca\") pod \"dc5a7f12-8474-49fa-b5ba-b3db497c0483\" (UID: \"dc5a7f12-8474-49fa-b5ba-b3db497c0483\") " Oct 30 21:38:47.145280 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:47.145166 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc5a7f12-8474-49fa-b5ba-b3db497c0483-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc5a7f12-8474-49fa-b5ba-b3db497c0483-maas-api-openshift-ca\") pod \"dc5a7f12-8474-49fa-b5ba-b3db497c0483\" (UID: \"dc5a7f12-8474-49fa-b5ba-b3db497c0483\") " Oct 30 21:38:47.145280 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:47.145205 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc5a7f12-8474-49fa-b5ba-b3db497c0483-kube-api-access-tfhwr\" (UniqueName: \"kubernetes.io/projected/dc5a7f12-8474-49fa-b5ba-b3db497c0483-kube-api-access-tfhwr\") pod \"dc5a7f12-8474-49fa-b5ba-b3db497c0483\" (UID: \"dc5a7f12-8474-49fa-b5ba-b3db497c0483\") " Oct 30 21:38:47.145601 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:47.145577 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc5a7f12-8474-49fa-b5ba-b3db497c0483-maas-api-openshift-ca" pod "dc5a7f12-8474-49fa-b5ba-b3db497c0483" (UID: "dc5a7f12-8474-49fa-b5ba-b3db497c0483"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:47.147326 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:47.147302 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc5a7f12-8474-49fa-b5ba-b3db497c0483-maas-api-xks-ca" pod "dc5a7f12-8474-49fa-b5ba-b3db497c0483" (UID: "dc5a7f12-8474-49fa-b5ba-b3db497c0483"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:38:47.147437 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:47.147372 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc5a7f12-8474-49fa-b5ba-b3db497c0483-kube-api-access-tfhwr" pod "dc5a7f12-8474-49fa-b5ba-b3db497c0483" (UID: "dc5a7f12-8474-49fa-b5ba-b3db497c0483"). InnerVolumeSpecName "kube-api-access-tfhwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:47.246522 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:47.246487 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc5a7f12-8474-49fa-b5ba-b3db497c0483-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:38:47.246522 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:47.246519 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfhwr\" (UniqueName: \"kubernetes.io/projected/dc5a7f12-8474-49fa-b5ba-b3db497c0483-kube-api-access-tfhwr\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:38:47.246522 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:47.246530 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc5a7f12-8474-49fa-b5ba-b3db497c0483-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:38:47.947249 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:47.947214 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7378b486-6p5r9" event={"ID":"dc5a7f12-8474-49fa-b5ba-b3db497c0483","Type":"ContainerDied","Data":"352f878fee2318e0f6c0d5de05c76a598ca15a8c33ced15cd36a83217e857fc8"} Oct 30 21:38:47.947420 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:47.947258 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7378b486-6p5r9" Oct 30 21:38:47.947420 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:47.947263 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="352f878fee2318e0f6c0d5de05c76a598ca15a8c33ced15cd36a83217e857fc8" Oct 30 21:38:49.615282 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:49.615246 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" Oct 30 21:38:49.627709 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:49.627687 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" Oct 30 21:38:49.684723 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:38:49.684630 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:38:49.973063 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:38:49.972980 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" Oct 30 21:39:08.980068 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:08.980032 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9"] Oct 30 21:39:08.980566 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:08.980527 4560 cpu_manager.go:414] "containerMap: removing container" podUID="dc5a7f12-8474-49fa-b5ba-b3db497c0483" containerName="revoke-keys" Oct 30 21:39:08.980566 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:08.980547 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc5a7f12-8474-49fa-b5ba-b3db497c0483" containerName="revoke-keys" Oct 30 21:39:08.980701 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:08.980686 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc5a7f12-8474-49fa-b5ba-b3db497c0483" containerName="revoke-keys" Oct 30 21:39:08.983419 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:08.983398 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" Oct 30 21:39:09.051669 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:09.051631 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a59a0fc3-8316-47eb-a42e-552d202b1ab7-model-cache\") pod \"premium-w0-afbe5b-kserve-5c568776bf-6wqw9\" (UID: \"a59a0fc3-8316-47eb-a42e-552d202b1ab7\") " pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" Oct 30 21:39:09.051839 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:09.051678 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a59a0fc3-8316-47eb-a42e-552d202b1ab7-tmp-dir\") pod \"premium-w0-afbe5b-kserve-5c568776bf-6wqw9\" (UID: \"a59a0fc3-8316-47eb-a42e-552d202b1ab7\") " pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" Oct 30 21:39:09.051839 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:09.051811 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgqnf\" (UniqueName: \"kubernetes.io/projected/a59a0fc3-8316-47eb-a42e-552d202b1ab7-kube-api-access-cgqnf\") pod \"premium-w0-afbe5b-kserve-5c568776bf-6wqw9\" (UID: \"a59a0fc3-8316-47eb-a42e-552d202b1ab7\") " pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" Oct 30 21:39:09.051950 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:09.051889 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a59a0fc3-8316-47eb-a42e-552d202b1ab7-tls-certs\") pod \"premium-w0-afbe5b-kserve-5c568776bf-6wqw9\" (UID: \"a59a0fc3-8316-47eb-a42e-552d202b1ab7\") " pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" Oct 30 21:39:09.052057 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:09.052042 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a59a0fc3-8316-47eb-a42e-552d202b1ab7-dshm\") pod \"premium-w0-afbe5b-kserve-5c568776bf-6wqw9\" (UID: \"a59a0fc3-8316-47eb-a42e-552d202b1ab7\") " pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" Oct 30 21:39:09.052119 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:09.052074 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a59a0fc3-8316-47eb-a42e-552d202b1ab7-home\") pod \"premium-w0-afbe5b-kserve-5c568776bf-6wqw9\" (UID: \"a59a0fc3-8316-47eb-a42e-552d202b1ab7\") " pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" Oct 30 21:39:09.079437 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:09.079411 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-afbe5b\"/\"premium-w0-afbe5b-kserve-self-signed-certs\"" Oct 30 21:39:09.081131 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:09.081111 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9"] Oct 30 21:39:09.152630 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:09.152590 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a59a0fc3-8316-47eb-a42e-552d202b1ab7-dshm\") pod \"premium-w0-afbe5b-kserve-5c568776bf-6wqw9\" (UID: \"a59a0fc3-8316-47eb-a42e-552d202b1ab7\") " pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" Oct 30 21:39:09.152630 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:09.152633 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a59a0fc3-8316-47eb-a42e-552d202b1ab7-home\") pod \"premium-w0-afbe5b-kserve-5c568776bf-6wqw9\" (UID: \"a59a0fc3-8316-47eb-a42e-552d202b1ab7\") " pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" Oct 30 21:39:09.152877 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:09.152681 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a59a0fc3-8316-47eb-a42e-552d202b1ab7-model-cache\") pod \"premium-w0-afbe5b-kserve-5c568776bf-6wqw9\" (UID: \"a59a0fc3-8316-47eb-a42e-552d202b1ab7\") " pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" Oct 30 21:39:09.152877 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:09.152714 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a59a0fc3-8316-47eb-a42e-552d202b1ab7-tmp-dir\") pod \"premium-w0-afbe5b-kserve-5c568776bf-6wqw9\" (UID: \"a59a0fc3-8316-47eb-a42e-552d202b1ab7\") " pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" Oct 30 21:39:09.152877 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:09.152750 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgqnf\" (UniqueName: \"kubernetes.io/projected/a59a0fc3-8316-47eb-a42e-552d202b1ab7-kube-api-access-cgqnf\") pod \"premium-w0-afbe5b-kserve-5c568776bf-6wqw9\" (UID: \"a59a0fc3-8316-47eb-a42e-552d202b1ab7\") " pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" Oct 30 21:39:09.152877 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:09.152785 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a59a0fc3-8316-47eb-a42e-552d202b1ab7-tls-certs\") pod \"premium-w0-afbe5b-kserve-5c568776bf-6wqw9\" (UID: \"a59a0fc3-8316-47eb-a42e-552d202b1ab7\") " pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" Oct 30 21:39:09.153139 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:09.153110 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a59a0fc3-8316-47eb-a42e-552d202b1ab7-home\") pod \"premium-w0-afbe5b-kserve-5c568776bf-6wqw9\" (UID: \"a59a0fc3-8316-47eb-a42e-552d202b1ab7\") " pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" Oct 30 21:39:09.153269 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:09.153130 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a59a0fc3-8316-47eb-a42e-552d202b1ab7-model-cache\") pod \"premium-w0-afbe5b-kserve-5c568776bf-6wqw9\" (UID: \"a59a0fc3-8316-47eb-a42e-552d202b1ab7\") " pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" Oct 30 21:39:09.153269 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:09.153227 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a59a0fc3-8316-47eb-a42e-552d202b1ab7-tmp-dir\") pod \"premium-w0-afbe5b-kserve-5c568776bf-6wqw9\" (UID: \"a59a0fc3-8316-47eb-a42e-552d202b1ab7\") " pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" Oct 30 21:39:09.155016 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:09.154995 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a59a0fc3-8316-47eb-a42e-552d202b1ab7-dshm\") pod \"premium-w0-afbe5b-kserve-5c568776bf-6wqw9\" (UID: \"a59a0fc3-8316-47eb-a42e-552d202b1ab7\") " pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" Oct 30 21:39:09.155320 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:09.155305 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a59a0fc3-8316-47eb-a42e-552d202b1ab7-tls-certs\") pod \"premium-w0-afbe5b-kserve-5c568776bf-6wqw9\" (UID: \"a59a0fc3-8316-47eb-a42e-552d202b1ab7\") " pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" Oct 30 21:39:09.210530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:09.210499 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgqnf\" (UniqueName: \"kubernetes.io/projected/a59a0fc3-8316-47eb-a42e-552d202b1ab7-kube-api-access-cgqnf\") pod \"premium-w0-afbe5b-kserve-5c568776bf-6wqw9\" (UID: \"a59a0fc3-8316-47eb-a42e-552d202b1ab7\") " pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" Oct 30 21:39:09.294566 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:09.294481 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" Oct 30 21:39:09.677916 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:09.677840 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35f9b9ef-8xlqp"] Oct 30 21:39:09.686835 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:09.686813 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f9b9ef-8xlqp" Oct 30 21:39:09.687737 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:09.687715 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9"] Oct 30 21:39:09.693991 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:39:09.693969 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda59a0fc3_8316_47eb_a42e_552d202b1ab7.slice/crio-e08b506f3f7481b6489a9d788ac7c848e4bb23605224d74da0380c309730f764 WatchSource:0}: Error finding container e08b506f3f7481b6489a9d788ac7c848e4bb23605224d74da0380c309730f764: Status 404 returned error can't find the container with id e08b506f3f7481b6489a9d788ac7c848e4bb23605224d74da0380c309730f764 Oct 30 21:39:09.736229 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:09.736197 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35f9b9ef-8xlqp"] Oct 30 21:39:09.736387 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:09.736370 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:39:09.736459 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:09.736392 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:39:09.736459 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:09.736405 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:39:09.760485 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:09.760457 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwjgf\" (UniqueName: \"kubernetes.io/projected/3f104d1a-2722-4dc1-be18-12a7786b22c9-kube-api-access-fwjgf\") pod \"maas-api-revoke-sub-35f9b9ef-8xlqp\" (UID: \"3f104d1a-2722-4dc1-be18-12a7786b22c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f9b9ef-8xlqp" Oct 30 21:39:09.760629 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:09.760492 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f104d1a-2722-4dc1-be18-12a7786b22c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35f9b9ef-8xlqp\" (UID: \"3f104d1a-2722-4dc1-be18-12a7786b22c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f9b9ef-8xlqp" Oct 30 21:39:09.760629 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:09.760526 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f104d1a-2722-4dc1-be18-12a7786b22c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35f9b9ef-8xlqp\" (UID: \"3f104d1a-2722-4dc1-be18-12a7786b22c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f9b9ef-8xlqp" Oct 30 21:39:09.780823 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:09.780799 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:39:09.861104 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:09.861070 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwjgf\" (UniqueName: \"kubernetes.io/projected/3f104d1a-2722-4dc1-be18-12a7786b22c9-kube-api-access-fwjgf\") pod \"maas-api-revoke-sub-35f9b9ef-8xlqp\" (UID: \"3f104d1a-2722-4dc1-be18-12a7786b22c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f9b9ef-8xlqp" Oct 30 21:39:09.861104 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:09.861105 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f104d1a-2722-4dc1-be18-12a7786b22c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35f9b9ef-8xlqp\" (UID: \"3f104d1a-2722-4dc1-be18-12a7786b22c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f9b9ef-8xlqp" Oct 30 21:39:09.861326 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:09.861236 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f104d1a-2722-4dc1-be18-12a7786b22c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35f9b9ef-8xlqp\" (UID: \"3f104d1a-2722-4dc1-be18-12a7786b22c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f9b9ef-8xlqp" Oct 30 21:39:09.861883 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:09.861855 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f104d1a-2722-4dc1-be18-12a7786b22c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35f9b9ef-8xlqp\" (UID: \"3f104d1a-2722-4dc1-be18-12a7786b22c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f9b9ef-8xlqp" Oct 30 21:39:09.863363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:09.863347 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f104d1a-2722-4dc1-be18-12a7786b22c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35f9b9ef-8xlqp\" (UID: \"3f104d1a-2722-4dc1-be18-12a7786b22c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f9b9ef-8xlqp" Oct 30 21:39:09.919512 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:09.919482 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwjgf\" (UniqueName: \"kubernetes.io/projected/3f104d1a-2722-4dc1-be18-12a7786b22c9-kube-api-access-fwjgf\") pod \"maas-api-revoke-sub-35f9b9ef-8xlqp\" (UID: \"3f104d1a-2722-4dc1-be18-12a7786b22c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f9b9ef-8xlqp" Oct 30 21:39:10.001593 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:10.001566 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f9b9ef-8xlqp" Oct 30 21:39:10.053020 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:10.052731 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" event={"ID":"a59a0fc3-8316-47eb-a42e-552d202b1ab7","Type":"ContainerStarted","Data":"3d9ac6e9d0d2264fe8c1ecfa3b11164581832dce26b6b645f05d81fb438adca5"} Oct 30 21:39:10.053020 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:10.052790 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" event={"ID":"a59a0fc3-8316-47eb-a42e-552d202b1ab7","Type":"ContainerStarted","Data":"e08b506f3f7481b6489a9d788ac7c848e4bb23605224d74da0380c309730f764"} Oct 30 21:39:10.164989 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:10.164935 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" podStartSLOduration=2.164922681 podStartE2EDuration="2.164922681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:10.162710056 +0000 UTC m=+1525.717409833" watchObservedRunningTime="2026-10-30 21:39:10.164922681 +0000 UTC m=+1525.719622458" Oct 30 21:39:10.295742 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:10.295364 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" Oct 30 21:39:10.371738 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:10.371703 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35f9b9ef-8xlqp"] Oct 30 21:39:10.378601 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:39:10.378568 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f104d1a_2722_4dc1_be18_12a7786b22c9.slice/crio-6053e452b784e7f25b2526634770d54a3b700abacbe01c4d0f31f679cba3577a WatchSource:0}: Error finding container 6053e452b784e7f25b2526634770d54a3b700abacbe01c4d0f31f679cba3577a: Status 404 returned error can't find the container with id 6053e452b784e7f25b2526634770d54a3b700abacbe01c4d0f31f679cba3577a Oct 30 21:39:10.693480 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:10.693453 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:39:11.059216 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:11.059180 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f9b9ef-8xlqp" event={"ID":"3f104d1a-2722-4dc1-be18-12a7786b22c9","Type":"ContainerStarted","Data":"11e82a0aa5b0e62b53cc9637f57c0b826bcbd1d0d344fbefea56084fc882295e"} Oct 30 21:39:11.059634 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:11.059266 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f9b9ef-8xlqp" event={"ID":"3f104d1a-2722-4dc1-be18-12a7786b22c9","Type":"ContainerStarted","Data":"6053e452b784e7f25b2526634770d54a3b700abacbe01c4d0f31f679cba3577a"} Oct 30 21:39:11.169794 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:11.169739 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f9b9ef-8xlqp" podStartSLOduration=2.169725032 podStartE2EDuration="2.169725032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:11.168704482 +0000 UTC m=+1526.723404259" watchObservedRunningTime="2026-10-30 21:39:11.169725032 +0000 UTC m=+1526.724424808" Oct 30 21:39:12.065573 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:12.065537 4560 generic.go:356] "Generic (PLEG): container finished" podID="3f104d1a-2722-4dc1-be18-12a7786b22c9" containerID="11e82a0aa5b0e62b53cc9637f57c0b826bcbd1d0d344fbefea56084fc882295e" exitCode=0 Oct 30 21:39:12.065964 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:12.065626 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f9b9ef-8xlqp" event={"ID":"3f104d1a-2722-4dc1-be18-12a7786b22c9","Type":"ContainerDied","Data":"11e82a0aa5b0e62b53cc9637f57c0b826bcbd1d0d344fbefea56084fc882295e"} Oct 30 21:39:12.853747 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:12.853712 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddda40cb-sb7df"] Oct 30 21:39:12.857207 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:12.857185 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddda40cb-sb7df" Oct 30 21:39:12.911927 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:12.911896 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddda40cb-sb7df"] Oct 30 21:39:12.991192 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:12.991147 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdmsf\" (UniqueName: \"kubernetes.io/projected/678fb080-b0c0-4967-9943-409845f0b31c-kube-api-access-mdmsf\") pod \"maas-api-revoke-sub-ddda40cb-sb7df\" (UID: \"678fb080-b0c0-4967-9943-409845f0b31c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddda40cb-sb7df" Oct 30 21:39:12.991192 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:12.991196 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/678fb080-b0c0-4967-9943-409845f0b31c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddda40cb-sb7df\" (UID: \"678fb080-b0c0-4967-9943-409845f0b31c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddda40cb-sb7df" Oct 30 21:39:12.991480 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:12.991454 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/678fb080-b0c0-4967-9943-409845f0b31c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddda40cb-sb7df\" (UID: \"678fb080-b0c0-4967-9943-409845f0b31c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddda40cb-sb7df" Oct 30 21:39:13.092204 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:13.092162 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/678fb080-b0c0-4967-9943-409845f0b31c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddda40cb-sb7df\" (UID: \"678fb080-b0c0-4967-9943-409845f0b31c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddda40cb-sb7df" Oct 30 21:39:13.092204 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:13.092199 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdmsf\" (UniqueName: \"kubernetes.io/projected/678fb080-b0c0-4967-9943-409845f0b31c-kube-api-access-mdmsf\") pod \"maas-api-revoke-sub-ddda40cb-sb7df\" (UID: \"678fb080-b0c0-4967-9943-409845f0b31c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddda40cb-sb7df" Oct 30 21:39:13.092729 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:13.092217 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/678fb080-b0c0-4967-9943-409845f0b31c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddda40cb-sb7df\" (UID: \"678fb080-b0c0-4967-9943-409845f0b31c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddda40cb-sb7df" Oct 30 21:39:13.092831 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:13.092808 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/678fb080-b0c0-4967-9943-409845f0b31c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddda40cb-sb7df\" (UID: \"678fb080-b0c0-4967-9943-409845f0b31c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddda40cb-sb7df" Oct 30 21:39:13.094653 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:13.094627 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/678fb080-b0c0-4967-9943-409845f0b31c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddda40cb-sb7df\" (UID: \"678fb080-b0c0-4967-9943-409845f0b31c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddda40cb-sb7df" Oct 30 21:39:13.164649 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:13.164620 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdmsf\" (UniqueName: \"kubernetes.io/projected/678fb080-b0c0-4967-9943-409845f0b31c-kube-api-access-mdmsf\") pod \"maas-api-revoke-sub-ddda40cb-sb7df\" (UID: \"678fb080-b0c0-4967-9943-409845f0b31c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddda40cb-sb7df" Oct 30 21:39:13.170420 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:13.170387 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddda40cb-sb7df" Oct 30 21:39:13.204710 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:13.204685 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f9b9ef-8xlqp" Oct 30 21:39:13.294922 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:13.294894 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f104d1a-2722-4dc1-be18-12a7786b22c9-kube-api-access-fwjgf\" (UniqueName: \"kubernetes.io/projected/3f104d1a-2722-4dc1-be18-12a7786b22c9-kube-api-access-fwjgf\") pod \"3f104d1a-2722-4dc1-be18-12a7786b22c9\" (UID: \"3f104d1a-2722-4dc1-be18-12a7786b22c9\") " Oct 30 21:39:13.295085 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:13.294981 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f104d1a-2722-4dc1-be18-12a7786b22c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f104d1a-2722-4dc1-be18-12a7786b22c9-maas-api-openshift-ca\") pod \"3f104d1a-2722-4dc1-be18-12a7786b22c9\" (UID: \"3f104d1a-2722-4dc1-be18-12a7786b22c9\") " Oct 30 21:39:13.295133 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:13.295103 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f104d1a-2722-4dc1-be18-12a7786b22c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f104d1a-2722-4dc1-be18-12a7786b22c9-maas-api-xks-ca\") pod \"3f104d1a-2722-4dc1-be18-12a7786b22c9\" (UID: \"3f104d1a-2722-4dc1-be18-12a7786b22c9\") " Oct 30 21:39:13.295366 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:13.295315 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f104d1a-2722-4dc1-be18-12a7786b22c9-maas-api-openshift-ca" pod "3f104d1a-2722-4dc1-be18-12a7786b22c9" (UID: "3f104d1a-2722-4dc1-be18-12a7786b22c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:13.297024 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:13.296999 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f104d1a-2722-4dc1-be18-12a7786b22c9-maas-api-xks-ca" pod "3f104d1a-2722-4dc1-be18-12a7786b22c9" (UID: "3f104d1a-2722-4dc1-be18-12a7786b22c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:13.297102 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:13.297083 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f104d1a-2722-4dc1-be18-12a7786b22c9-kube-api-access-fwjgf" pod "3f104d1a-2722-4dc1-be18-12a7786b22c9" (UID: "3f104d1a-2722-4dc1-be18-12a7786b22c9"). InnerVolumeSpecName "kube-api-access-fwjgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:13.396660 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:13.396588 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f104d1a-2722-4dc1-be18-12a7786b22c9-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:13.396660 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:13.396613 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwjgf\" (UniqueName: \"kubernetes.io/projected/3f104d1a-2722-4dc1-be18-12a7786b22c9-kube-api-access-fwjgf\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:13.396660 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:13.396623 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f104d1a-2722-4dc1-be18-12a7786b22c9-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:13.543848 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:13.543810 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddda40cb-sb7df"] Oct 30 21:39:13.550580 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:39:13.550554 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod678fb080_b0c0_4967_9943_409845f0b31c.slice/crio-a3d23dc7b55d3d2d64dffbccd327fbcb2b4d1fd122a49cf7b38e21c321210b02 WatchSource:0}: Error finding container a3d23dc7b55d3d2d64dffbccd327fbcb2b4d1fd122a49cf7b38e21c321210b02: Status 404 returned error can't find the container with id a3d23dc7b55d3d2d64dffbccd327fbcb2b4d1fd122a49cf7b38e21c321210b02 Oct 30 21:39:14.076104 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:14.076017 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f9b9ef-8xlqp" Oct 30 21:39:14.076104 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:14.076056 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f9b9ef-8xlqp" event={"ID":"3f104d1a-2722-4dc1-be18-12a7786b22c9","Type":"ContainerDied","Data":"6053e452b784e7f25b2526634770d54a3b700abacbe01c4d0f31f679cba3577a"} Oct 30 21:39:14.076284 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:14.076113 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6053e452b784e7f25b2526634770d54a3b700abacbe01c4d0f31f679cba3577a" Oct 30 21:39:14.077717 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:14.077689 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddda40cb-sb7df" event={"ID":"678fb080-b0c0-4967-9943-409845f0b31c","Type":"ContainerStarted","Data":"b6ef102f57beccaec3966804bfb7d420ed218f53efa8dc4f2a8111627bf0748c"} Oct 30 21:39:14.077825 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:14.077740 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddda40cb-sb7df" event={"ID":"678fb080-b0c0-4967-9943-409845f0b31c","Type":"ContainerStarted","Data":"a3d23dc7b55d3d2d64dffbccd327fbcb2b4d1fd122a49cf7b38e21c321210b02"} Oct 30 21:39:14.188254 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:14.188198 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddda40cb-sb7df" podStartSLOduration=2.188183643 podStartE2EDuration="2.188183643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:14.187136037 +0000 UTC m=+1529.741835816" watchObservedRunningTime="2026-10-30 21:39:14.188183643 +0000 UTC m=+1529.742883420" Oct 30 21:39:15.083487 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:15.083455 4560 generic.go:356] "Generic (PLEG): container finished" podID="678fb080-b0c0-4967-9943-409845f0b31c" containerID="b6ef102f57beccaec3966804bfb7d420ed218f53efa8dc4f2a8111627bf0748c" exitCode=0 Oct 30 21:39:15.083663 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:15.083536 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddda40cb-sb7df" event={"ID":"678fb080-b0c0-4967-9943-409845f0b31c","Type":"ContainerDied","Data":"b6ef102f57beccaec3966804bfb7d420ed218f53efa8dc4f2a8111627bf0748c"} Oct 30 21:39:16.230831 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:16.230805 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddda40cb-sb7df" Oct 30 21:39:16.323794 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:16.323763 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/678fb080-b0c0-4967-9943-409845f0b31c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/678fb080-b0c0-4967-9943-409845f0b31c-maas-api-xks-ca\") pod \"678fb080-b0c0-4967-9943-409845f0b31c\" (UID: \"678fb080-b0c0-4967-9943-409845f0b31c\") " Oct 30 21:39:16.323956 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:16.323808 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/678fb080-b0c0-4967-9943-409845f0b31c-kube-api-access-mdmsf\" (UniqueName: \"kubernetes.io/projected/678fb080-b0c0-4967-9943-409845f0b31c-kube-api-access-mdmsf\") pod \"678fb080-b0c0-4967-9943-409845f0b31c\" (UID: \"678fb080-b0c0-4967-9943-409845f0b31c\") " Oct 30 21:39:16.323956 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:16.323889 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/678fb080-b0c0-4967-9943-409845f0b31c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/678fb080-b0c0-4967-9943-409845f0b31c-maas-api-openshift-ca\") pod \"678fb080-b0c0-4967-9943-409845f0b31c\" (UID: \"678fb080-b0c0-4967-9943-409845f0b31c\") " Oct 30 21:39:16.324304 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:16.324278 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/678fb080-b0c0-4967-9943-409845f0b31c-maas-api-openshift-ca" pod "678fb080-b0c0-4967-9943-409845f0b31c" (UID: "678fb080-b0c0-4967-9943-409845f0b31c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:16.325964 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:16.325944 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/678fb080-b0c0-4967-9943-409845f0b31c-maas-api-xks-ca" pod "678fb080-b0c0-4967-9943-409845f0b31c" (UID: "678fb080-b0c0-4967-9943-409845f0b31c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:16.326025 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:16.326009 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/678fb080-b0c0-4967-9943-409845f0b31c-kube-api-access-mdmsf" pod "678fb080-b0c0-4967-9943-409845f0b31c" (UID: "678fb080-b0c0-4967-9943-409845f0b31c"). InnerVolumeSpecName "kube-api-access-mdmsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:16.424901 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:16.424820 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/678fb080-b0c0-4967-9943-409845f0b31c-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:16.424901 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:16.424852 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/678fb080-b0c0-4967-9943-409845f0b31c-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:16.424901 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:16.424861 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdmsf\" (UniqueName: \"kubernetes.io/projected/678fb080-b0c0-4967-9943-409845f0b31c-kube-api-access-mdmsf\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:17.094725 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:17.094678 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddda40cb-sb7df" event={"ID":"678fb080-b0c0-4967-9943-409845f0b31c","Type":"ContainerDied","Data":"a3d23dc7b55d3d2d64dffbccd327fbcb2b4d1fd122a49cf7b38e21c321210b02"} Oct 30 21:39:17.094874 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:17.094741 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3d23dc7b55d3d2d64dffbccd327fbcb2b4d1fd122a49cf7b38e21c321210b02" Oct 30 21:39:17.094874 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:17.094743 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddda40cb-sb7df" Oct 30 21:39:19.295224 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:19.295183 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" Oct 30 21:39:19.308126 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:19.308099 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" Oct 30 21:39:19.686374 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:39:19.686271 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:39:20.120367 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:20.120322 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" Oct 30 21:39:25.370176 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:25.370123 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6861dafe-b5bfq"] Oct 30 21:39:25.370959 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:25.370934 4560 cpu_manager.go:414] "containerMap: removing container" podUID="678fb080-b0c0-4967-9943-409845f0b31c" containerName="revoke-keys" Oct 30 21:39:25.370959 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:25.370959 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="678fb080-b0c0-4967-9943-409845f0b31c" containerName="revoke-keys" Oct 30 21:39:25.371090 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:25.371010 4560 cpu_manager.go:414] "containerMap: removing container" podUID="3f104d1a-2722-4dc1-be18-12a7786b22c9" containerName="revoke-keys" Oct 30 21:39:25.371090 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:25.371019 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f104d1a-2722-4dc1-be18-12a7786b22c9" containerName="revoke-keys" Oct 30 21:39:25.371205 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:25.371192 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="678fb080-b0c0-4967-9943-409845f0b31c" containerName="revoke-keys" Oct 30 21:39:25.371251 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:25.371217 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f104d1a-2722-4dc1-be18-12a7786b22c9" containerName="revoke-keys" Oct 30 21:39:25.375306 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:25.375279 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6861dafe-b5bfq" Oct 30 21:39:25.410686 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:25.410647 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5mb5\" (UniqueName: \"kubernetes.io/projected/cc375f8c-fc71-46ed-84a1-58413a18eaac-kube-api-access-k5mb5\") pod \"maas-api-revoke-sub-6861dafe-b5bfq\" (UID: \"cc375f8c-fc71-46ed-84a1-58413a18eaac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6861dafe-b5bfq" Oct 30 21:39:25.410855 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:25.410725 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc375f8c-fc71-46ed-84a1-58413a18eaac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6861dafe-b5bfq\" (UID: \"cc375f8c-fc71-46ed-84a1-58413a18eaac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6861dafe-b5bfq" Oct 30 21:39:25.410855 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:25.410759 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc375f8c-fc71-46ed-84a1-58413a18eaac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6861dafe-b5bfq\" (UID: \"cc375f8c-fc71-46ed-84a1-58413a18eaac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6861dafe-b5bfq" Oct 30 21:39:25.427007 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:25.426976 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6861dafe-b5bfq"] Oct 30 21:39:25.471260 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:25.471226 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:39:25.471260 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:25.471243 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:39:25.471260 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:25.471246 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:39:25.472591 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:25.472573 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:39:25.511367 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:25.511323 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5mb5\" (UniqueName: \"kubernetes.io/projected/cc375f8c-fc71-46ed-84a1-58413a18eaac-kube-api-access-k5mb5\") pod \"maas-api-revoke-sub-6861dafe-b5bfq\" (UID: \"cc375f8c-fc71-46ed-84a1-58413a18eaac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6861dafe-b5bfq" Oct 30 21:39:25.511540 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:25.511413 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc375f8c-fc71-46ed-84a1-58413a18eaac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6861dafe-b5bfq\" (UID: \"cc375f8c-fc71-46ed-84a1-58413a18eaac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6861dafe-b5bfq" Oct 30 21:39:25.511540 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:25.511445 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc375f8c-fc71-46ed-84a1-58413a18eaac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6861dafe-b5bfq\" (UID: \"cc375f8c-fc71-46ed-84a1-58413a18eaac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6861dafe-b5bfq" Oct 30 21:39:25.512104 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:25.512082 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc375f8c-fc71-46ed-84a1-58413a18eaac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6861dafe-b5bfq\" (UID: \"cc375f8c-fc71-46ed-84a1-58413a18eaac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6861dafe-b5bfq" Oct 30 21:39:25.513776 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:25.513759 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc375f8c-fc71-46ed-84a1-58413a18eaac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6861dafe-b5bfq\" (UID: \"cc375f8c-fc71-46ed-84a1-58413a18eaac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6861dafe-b5bfq" Oct 30 21:39:25.569971 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:25.569928 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5mb5\" (UniqueName: \"kubernetes.io/projected/cc375f8c-fc71-46ed-84a1-58413a18eaac-kube-api-access-k5mb5\") pod \"maas-api-revoke-sub-6861dafe-b5bfq\" (UID: \"cc375f8c-fc71-46ed-84a1-58413a18eaac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6861dafe-b5bfq" Oct 30 21:39:25.688018 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:25.687922 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6861dafe-b5bfq" Oct 30 21:39:26.058646 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:26.058612 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6861dafe-b5bfq"] Oct 30 21:39:26.065154 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:39:26.065127 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc375f8c_fc71_46ed_84a1_58413a18eaac.slice/crio-7efbc268d08cc2407f8fbe7e899c5bd4816d8ced0863de37a187e9066dc56fcd WatchSource:0}: Error finding container 7efbc268d08cc2407f8fbe7e899c5bd4816d8ced0863de37a187e9066dc56fcd: Status 404 returned error can't find the container with id 7efbc268d08cc2407f8fbe7e899c5bd4816d8ced0863de37a187e9066dc56fcd Oct 30 21:39:26.138596 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:26.138563 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6861dafe-b5bfq" event={"ID":"cc375f8c-fc71-46ed-84a1-58413a18eaac","Type":"ContainerStarted","Data":"7efbc268d08cc2407f8fbe7e899c5bd4816d8ced0863de37a187e9066dc56fcd"} Oct 30 21:39:27.144693 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:27.144603 4560 generic.go:356] "Generic (PLEG): container finished" podID="cc375f8c-fc71-46ed-84a1-58413a18eaac" containerID="37b3f9c7df364cd509d38459eea6c04f135ff32c7c543c6203c3d53be99e4b7b" exitCode=0 Oct 30 21:39:27.145113 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:27.144688 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6861dafe-b5bfq" event={"ID":"cc375f8c-fc71-46ed-84a1-58413a18eaac","Type":"ContainerDied","Data":"37b3f9c7df364cd509d38459eea6c04f135ff32c7c543c6203c3d53be99e4b7b"} Oct 30 21:39:28.288765 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:28.288740 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6861dafe-b5bfq" Oct 30 21:39:28.348199 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:28.348166 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc375f8c-fc71-46ed-84a1-58413a18eaac-kube-api-access-k5mb5\" (UniqueName: \"kubernetes.io/projected/cc375f8c-fc71-46ed-84a1-58413a18eaac-kube-api-access-k5mb5\") pod \"cc375f8c-fc71-46ed-84a1-58413a18eaac\" (UID: \"cc375f8c-fc71-46ed-84a1-58413a18eaac\") " Oct 30 21:39:28.348407 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:28.348287 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc375f8c-fc71-46ed-84a1-58413a18eaac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc375f8c-fc71-46ed-84a1-58413a18eaac-maas-api-xks-ca\") pod \"cc375f8c-fc71-46ed-84a1-58413a18eaac\" (UID: \"cc375f8c-fc71-46ed-84a1-58413a18eaac\") " Oct 30 21:39:28.348407 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:28.348312 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc375f8c-fc71-46ed-84a1-58413a18eaac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc375f8c-fc71-46ed-84a1-58413a18eaac-maas-api-openshift-ca\") pod \"cc375f8c-fc71-46ed-84a1-58413a18eaac\" (UID: \"cc375f8c-fc71-46ed-84a1-58413a18eaac\") " Oct 30 21:39:28.348772 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:28.348744 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc375f8c-fc71-46ed-84a1-58413a18eaac-maas-api-openshift-ca" pod "cc375f8c-fc71-46ed-84a1-58413a18eaac" (UID: "cc375f8c-fc71-46ed-84a1-58413a18eaac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:28.350417 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:28.350393 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc375f8c-fc71-46ed-84a1-58413a18eaac-maas-api-xks-ca" pod "cc375f8c-fc71-46ed-84a1-58413a18eaac" (UID: "cc375f8c-fc71-46ed-84a1-58413a18eaac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:28.350500 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:28.350479 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc375f8c-fc71-46ed-84a1-58413a18eaac-kube-api-access-k5mb5" pod "cc375f8c-fc71-46ed-84a1-58413a18eaac" (UID: "cc375f8c-fc71-46ed-84a1-58413a18eaac"). InnerVolumeSpecName "kube-api-access-k5mb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:28.449894 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:28.449812 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc375f8c-fc71-46ed-84a1-58413a18eaac-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:28.449894 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:28.449840 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc375f8c-fc71-46ed-84a1-58413a18eaac-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:28.449894 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:28.449851 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5mb5\" (UniqueName: \"kubernetes.io/projected/cc375f8c-fc71-46ed-84a1-58413a18eaac-kube-api-access-k5mb5\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:29.154417 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:29.154390 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6861dafe-b5bfq" Oct 30 21:39:29.154417 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:29.154413 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6861dafe-b5bfq" event={"ID":"cc375f8c-fc71-46ed-84a1-58413a18eaac","Type":"ContainerDied","Data":"7efbc268d08cc2407f8fbe7e899c5bd4816d8ced0863de37a187e9066dc56fcd"} Oct 30 21:39:29.154620 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:29.154463 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7efbc268d08cc2407f8fbe7e899c5bd4816d8ced0863de37a187e9066dc56fcd" Oct 30 21:39:32.995537 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:32.995499 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf7cbcc7-8526x"] Oct 30 21:39:32.996010 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:32.995967 4560 cpu_manager.go:414] "containerMap: removing container" podUID="cc375f8c-fc71-46ed-84a1-58413a18eaac" containerName="revoke-keys" Oct 30 21:39:32.996010 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:32.995984 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc375f8c-fc71-46ed-84a1-58413a18eaac" containerName="revoke-keys" Oct 30 21:39:32.996152 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:32.996138 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc375f8c-fc71-46ed-84a1-58413a18eaac" containerName="revoke-keys" Oct 30 21:39:32.999490 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:32.999470 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7cbcc7-8526x" Oct 30 21:39:33.049173 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:33.049147 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:39:33.049304 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:33.049270 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:39:33.092617 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:33.092584 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf7cbcc7-8526x\" (UID: \"cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7cbcc7-8526x" Oct 30 21:39:33.092762 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:33.092636 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbjj6\" (UniqueName: \"kubernetes.io/projected/cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30-kube-api-access-dbjj6\") pod \"maas-api-revoke-sub-bf7cbcc7-8526x\" (UID: \"cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7cbcc7-8526x" Oct 30 21:39:33.092833 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:33.092802 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf7cbcc7-8526x\" (UID: \"cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7cbcc7-8526x" Oct 30 21:39:33.095553 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:33.095535 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:39:33.095632 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:33.095568 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:39:33.097326 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:33.097307 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf7cbcc7-8526x"] Oct 30 21:39:33.194015 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:33.193982 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbjj6\" (UniqueName: \"kubernetes.io/projected/cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30-kube-api-access-dbjj6\") pod \"maas-api-revoke-sub-bf7cbcc7-8526x\" (UID: \"cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7cbcc7-8526x" Oct 30 21:39:33.194174 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:33.194073 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf7cbcc7-8526x\" (UID: \"cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7cbcc7-8526x" Oct 30 21:39:33.194174 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:33.194121 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf7cbcc7-8526x\" (UID: \"cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7cbcc7-8526x" Oct 30 21:39:33.194732 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:33.194707 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf7cbcc7-8526x\" (UID: \"cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7cbcc7-8526x" Oct 30 21:39:33.196419 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:33.196400 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf7cbcc7-8526x\" (UID: \"cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7cbcc7-8526x" Oct 30 21:39:33.252092 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:33.252011 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbjj6\" (UniqueName: \"kubernetes.io/projected/cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30-kube-api-access-dbjj6\") pod \"maas-api-revoke-sub-bf7cbcc7-8526x\" (UID: \"cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7cbcc7-8526x" Oct 30 21:39:33.311235 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:33.311204 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7cbcc7-8526x" Oct 30 21:39:33.684225 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:33.684117 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf7cbcc7-8526x"] Oct 30 21:39:33.688625 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:39:33.688598 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcba9ba1d_2d8d_4f89_a7e0_a8febf52cb30.slice/crio-a87d44e30563eefbecb3de0b4d9c26cef26924741401aee5f17b2cbf13a4c3ec WatchSource:0}: Error finding container a87d44e30563eefbecb3de0b4d9c26cef26924741401aee5f17b2cbf13a4c3ec: Status 404 returned error can't find the container with id a87d44e30563eefbecb3de0b4d9c26cef26924741401aee5f17b2cbf13a4c3ec Oct 30 21:39:34.184584 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:34.184546 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7cbcc7-8526x" event={"ID":"cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30","Type":"ContainerStarted","Data":"8dc0a87edb830e988a276c29eec9b058dbc7127ef572fcc38dab4622b2bb801b"} Oct 30 21:39:34.185044 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:34.184620 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7cbcc7-8526x" event={"ID":"cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30","Type":"ContainerStarted","Data":"a87d44e30563eefbecb3de0b4d9c26cef26924741401aee5f17b2cbf13a4c3ec"} Oct 30 21:39:34.295355 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:34.295282 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7cbcc7-8526x" podStartSLOduration=2.295268263 podStartE2EDuration="2.295268263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:34.293728203 +0000 UTC m=+1549.848427980" watchObservedRunningTime="2026-10-30 21:39:34.295268263 +0000 UTC m=+1549.849968040" Oct 30 21:39:35.191208 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:35.191172 4560 generic.go:356] "Generic (PLEG): container finished" podID="cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30" containerID="8dc0a87edb830e988a276c29eec9b058dbc7127ef572fcc38dab4622b2bb801b" exitCode=0 Oct 30 21:39:35.191674 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:35.191241 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7cbcc7-8526x" event={"ID":"cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30","Type":"ContainerDied","Data":"8dc0a87edb830e988a276c29eec9b058dbc7127ef572fcc38dab4622b2bb801b"} Oct 30 21:39:36.323355 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:36.323321 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7cbcc7-8526x" Oct 30 21:39:36.423993 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:36.423965 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30-maas-api-xks-ca\") pod \"cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30\" (UID: \"cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30\") " Oct 30 21:39:36.424152 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:36.424056 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30-kube-api-access-dbjj6\" (UniqueName: \"kubernetes.io/projected/cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30-kube-api-access-dbjj6\") pod \"cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30\" (UID: \"cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30\") " Oct 30 21:39:36.424152 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:36.424078 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30-maas-api-openshift-ca\") pod \"cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30\" (UID: \"cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30\") " Oct 30 21:39:36.424516 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:36.424494 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30-maas-api-openshift-ca" pod "cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30" (UID: "cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:36.425956 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:36.425933 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30-maas-api-xks-ca" pod "cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30" (UID: "cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:36.426032 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:36.425979 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30-kube-api-access-dbjj6" pod "cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30" (UID: "cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30"). InnerVolumeSpecName "kube-api-access-dbjj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:36.525158 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:36.525128 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:36.525158 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:36.525154 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbjj6\" (UniqueName: \"kubernetes.io/projected/cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30-kube-api-access-dbjj6\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:36.525358 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:36.525167 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:37.202987 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:37.202952 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7cbcc7-8526x" event={"ID":"cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30","Type":"ContainerDied","Data":"a87d44e30563eefbecb3de0b4d9c26cef26924741401aee5f17b2cbf13a4c3ec"} Oct 30 21:39:37.202987 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:37.202977 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7cbcc7-8526x" Oct 30 21:39:37.203189 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:37.202999 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a87d44e30563eefbecb3de0b4d9c26cef26924741401aee5f17b2cbf13a4c3ec" Oct 30 21:39:43.778377 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:43.778318 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f0f1c9b-hqbbz"] Oct 30 21:39:43.778746 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:43.778711 4560 cpu_manager.go:414] "containerMap: removing container" podUID="cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30" containerName="revoke-keys" Oct 30 21:39:43.778746 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:43.778722 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30" containerName="revoke-keys" Oct 30 21:39:43.778822 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:43.778784 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30" containerName="revoke-keys" Oct 30 21:39:43.782057 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:43.782035 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0f1c9b-hqbbz" Oct 30 21:39:43.836791 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:43.836763 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f0f1c9b-hqbbz"] Oct 30 21:39:43.880740 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:43.880709 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:39:43.880740 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:43.880711 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:39:43.880740 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:43.880721 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:39:43.880940 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:43.880778 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:39:43.890189 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:43.890165 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d68e17d8-4657-412e-b1a8-ec1f5b8a04a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f0f1c9b-hqbbz\" (UID: \"d68e17d8-4657-412e-b1a8-ec1f5b8a04a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0f1c9b-hqbbz" Oct 30 21:39:43.890316 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:43.890221 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d68e17d8-4657-412e-b1a8-ec1f5b8a04a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f0f1c9b-hqbbz\" (UID: \"d68e17d8-4657-412e-b1a8-ec1f5b8a04a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0f1c9b-hqbbz" Oct 30 21:39:43.890316 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:43.890241 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxgc6\" (UniqueName: \"kubernetes.io/projected/d68e17d8-4657-412e-b1a8-ec1f5b8a04a1-kube-api-access-rxgc6\") pod \"maas-api-revoke-sub-4f0f1c9b-hqbbz\" (UID: \"d68e17d8-4657-412e-b1a8-ec1f5b8a04a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0f1c9b-hqbbz" Oct 30 21:39:43.991736 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:43.991703 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d68e17d8-4657-412e-b1a8-ec1f5b8a04a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f0f1c9b-hqbbz\" (UID: \"d68e17d8-4657-412e-b1a8-ec1f5b8a04a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0f1c9b-hqbbz" Oct 30 21:39:43.991905 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:43.991743 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxgc6\" (UniqueName: \"kubernetes.io/projected/d68e17d8-4657-412e-b1a8-ec1f5b8a04a1-kube-api-access-rxgc6\") pod \"maas-api-revoke-sub-4f0f1c9b-hqbbz\" (UID: \"d68e17d8-4657-412e-b1a8-ec1f5b8a04a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0f1c9b-hqbbz" Oct 30 21:39:43.991905 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:43.991813 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d68e17d8-4657-412e-b1a8-ec1f5b8a04a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f0f1c9b-hqbbz\" (UID: \"d68e17d8-4657-412e-b1a8-ec1f5b8a04a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0f1c9b-hqbbz" Oct 30 21:39:43.992560 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:43.992537 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d68e17d8-4657-412e-b1a8-ec1f5b8a04a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f0f1c9b-hqbbz\" (UID: \"d68e17d8-4657-412e-b1a8-ec1f5b8a04a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0f1c9b-hqbbz" Oct 30 21:39:43.994035 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:43.994018 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d68e17d8-4657-412e-b1a8-ec1f5b8a04a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f0f1c9b-hqbbz\" (UID: \"d68e17d8-4657-412e-b1a8-ec1f5b8a04a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0f1c9b-hqbbz" Oct 30 21:39:44.050799 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:44.050736 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxgc6\" (UniqueName: \"kubernetes.io/projected/d68e17d8-4657-412e-b1a8-ec1f5b8a04a1-kube-api-access-rxgc6\") pod \"maas-api-revoke-sub-4f0f1c9b-hqbbz\" (UID: \"d68e17d8-4657-412e-b1a8-ec1f5b8a04a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0f1c9b-hqbbz" Oct 30 21:39:44.094856 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:44.094832 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0f1c9b-hqbbz" Oct 30 21:39:44.463938 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:44.463847 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f0f1c9b-hqbbz"] Oct 30 21:39:44.470177 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:39:44.470145 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd68e17d8_4657_412e_b1a8_ec1f5b8a04a1.slice/crio-4fb7c6a88d3e8bc0f98ef23b7cbadbe7e270d957f5305d6b506558fd159b3c6a WatchSource:0}: Error finding container 4fb7c6a88d3e8bc0f98ef23b7cbadbe7e270d957f5305d6b506558fd159b3c6a: Status 404 returned error can't find the container with id 4fb7c6a88d3e8bc0f98ef23b7cbadbe7e270d957f5305d6b506558fd159b3c6a Oct 30 21:39:45.241121 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:45.241084 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0f1c9b-hqbbz" event={"ID":"d68e17d8-4657-412e-b1a8-ec1f5b8a04a1","Type":"ContainerStarted","Data":"0879e2e115dd38598bc8e53662342d20ac703890bcc95945b362c9ac1aa41667"} Oct 30 21:39:45.241509 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:45.241136 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0f1c9b-hqbbz" event={"ID":"d68e17d8-4657-412e-b1a8-ec1f5b8a04a1","Type":"ContainerStarted","Data":"4fb7c6a88d3e8bc0f98ef23b7cbadbe7e270d957f5305d6b506558fd159b3c6a"} Oct 30 21:39:45.350910 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:45.350859 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0f1c9b-hqbbz" podStartSLOduration=2.350844131 podStartE2EDuration="2.350844131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:45.349988456 +0000 UTC m=+1560.904688235" watchObservedRunningTime="2026-10-30 21:39:45.350844131 +0000 UTC m=+1560.905543908" Oct 30 21:39:46.247708 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:46.247671 4560 generic.go:356] "Generic (PLEG): container finished" podID="d68e17d8-4657-412e-b1a8-ec1f5b8a04a1" containerID="0879e2e115dd38598bc8e53662342d20ac703890bcc95945b362c9ac1aa41667" exitCode=0 Oct 30 21:39:46.248129 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:46.247764 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0f1c9b-hqbbz" event={"ID":"d68e17d8-4657-412e-b1a8-ec1f5b8a04a1","Type":"ContainerDied","Data":"0879e2e115dd38598bc8e53662342d20ac703890bcc95945b362c9ac1aa41667"} Oct 30 21:39:47.404760 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:47.404735 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0f1c9b-hqbbz" Oct 30 21:39:47.525354 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:47.525259 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d68e17d8-4657-412e-b1a8-ec1f5b8a04a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d68e17d8-4657-412e-b1a8-ec1f5b8a04a1-maas-api-openshift-ca\") pod \"d68e17d8-4657-412e-b1a8-ec1f5b8a04a1\" (UID: \"d68e17d8-4657-412e-b1a8-ec1f5b8a04a1\") " Oct 30 21:39:47.525354 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:47.525325 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d68e17d8-4657-412e-b1a8-ec1f5b8a04a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d68e17d8-4657-412e-b1a8-ec1f5b8a04a1-maas-api-xks-ca\") pod \"d68e17d8-4657-412e-b1a8-ec1f5b8a04a1\" (UID: \"d68e17d8-4657-412e-b1a8-ec1f5b8a04a1\") " Oct 30 21:39:47.525612 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:47.525411 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d68e17d8-4657-412e-b1a8-ec1f5b8a04a1-kube-api-access-rxgc6\" (UniqueName: \"kubernetes.io/projected/d68e17d8-4657-412e-b1a8-ec1f5b8a04a1-kube-api-access-rxgc6\") pod \"d68e17d8-4657-412e-b1a8-ec1f5b8a04a1\" (UID: \"d68e17d8-4657-412e-b1a8-ec1f5b8a04a1\") " Oct 30 21:39:47.525670 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:47.525610 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d68e17d8-4657-412e-b1a8-ec1f5b8a04a1-maas-api-openshift-ca" pod "d68e17d8-4657-412e-b1a8-ec1f5b8a04a1" (UID: "d68e17d8-4657-412e-b1a8-ec1f5b8a04a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:47.525725 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:47.525705 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d68e17d8-4657-412e-b1a8-ec1f5b8a04a1-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:47.527539 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:47.527511 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d68e17d8-4657-412e-b1a8-ec1f5b8a04a1-maas-api-xks-ca" pod "d68e17d8-4657-412e-b1a8-ec1f5b8a04a1" (UID: "d68e17d8-4657-412e-b1a8-ec1f5b8a04a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:47.527539 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:47.527517 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d68e17d8-4657-412e-b1a8-ec1f5b8a04a1-kube-api-access-rxgc6" pod "d68e17d8-4657-412e-b1a8-ec1f5b8a04a1" (UID: "d68e17d8-4657-412e-b1a8-ec1f5b8a04a1"). InnerVolumeSpecName "kube-api-access-rxgc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:47.626927 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:47.626891 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d68e17d8-4657-412e-b1a8-ec1f5b8a04a1-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:47.626927 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:47.626924 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxgc6\" (UniqueName: \"kubernetes.io/projected/d68e17d8-4657-412e-b1a8-ec1f5b8a04a1-kube-api-access-rxgc6\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:48.258261 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:48.258228 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0f1c9b-hqbbz" Oct 30 21:39:48.258261 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:48.258257 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0f1c9b-hqbbz" event={"ID":"d68e17d8-4657-412e-b1a8-ec1f5b8a04a1","Type":"ContainerDied","Data":"4fb7c6a88d3e8bc0f98ef23b7cbadbe7e270d957f5305d6b506558fd159b3c6a"} Oct 30 21:39:48.258495 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:48.258329 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fb7c6a88d3e8bc0f98ef23b7cbadbe7e270d957f5305d6b506558fd159b3c6a" Oct 30 21:39:49.686529 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:39:49.686492 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:39:53.619731 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:53.619699 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:39:58.385133 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:58.385055 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a7b4e06-xv9vg"] Oct 30 21:39:58.385525 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:58.385475 4560 cpu_manager.go:414] "containerMap: removing container" podUID="d68e17d8-4657-412e-b1a8-ec1f5b8a04a1" containerName="revoke-keys" Oct 30 21:39:58.385525 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:58.385488 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d68e17d8-4657-412e-b1a8-ec1f5b8a04a1" containerName="revoke-keys" Oct 30 21:39:58.385603 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:58.385555 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="d68e17d8-4657-412e-b1a8-ec1f5b8a04a1" containerName="revoke-keys" Oct 30 21:39:58.388632 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:58.388616 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a7b4e06-xv9vg" Oct 30 21:39:58.437655 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:58.437614 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:39:58.437655 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:58.437613 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:39:58.483956 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:58.483925 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:39:58.484119 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:58.483972 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:39:58.485980 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:58.485961 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a7b4e06-xv9vg"] Oct 30 21:39:58.529968 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:58.529939 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30312af-f1f2-40e0-b630-db46b93837eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a7b4e06-xv9vg\" (UID: \"d30312af-f1f2-40e0-b630-db46b93837eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a7b4e06-xv9vg" Oct 30 21:39:58.530110 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:58.530077 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4pk9\" (UniqueName: \"kubernetes.io/projected/d30312af-f1f2-40e0-b630-db46b93837eb-kube-api-access-d4pk9\") pod \"maas-api-revoke-sub-1a7b4e06-xv9vg\" (UID: \"d30312af-f1f2-40e0-b630-db46b93837eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a7b4e06-xv9vg" Oct 30 21:39:58.530110 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:58.530105 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30312af-f1f2-40e0-b630-db46b93837eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a7b4e06-xv9vg\" (UID: \"d30312af-f1f2-40e0-b630-db46b93837eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a7b4e06-xv9vg" Oct 30 21:39:58.630642 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:58.630604 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4pk9\" (UniqueName: \"kubernetes.io/projected/d30312af-f1f2-40e0-b630-db46b93837eb-kube-api-access-d4pk9\") pod \"maas-api-revoke-sub-1a7b4e06-xv9vg\" (UID: \"d30312af-f1f2-40e0-b630-db46b93837eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a7b4e06-xv9vg" Oct 30 21:39:58.630642 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:58.630647 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30312af-f1f2-40e0-b630-db46b93837eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a7b4e06-xv9vg\" (UID: \"d30312af-f1f2-40e0-b630-db46b93837eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a7b4e06-xv9vg" Oct 30 21:39:58.630856 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:58.630681 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30312af-f1f2-40e0-b630-db46b93837eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a7b4e06-xv9vg\" (UID: \"d30312af-f1f2-40e0-b630-db46b93837eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a7b4e06-xv9vg" Oct 30 21:39:58.631367 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:58.631315 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30312af-f1f2-40e0-b630-db46b93837eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a7b4e06-xv9vg\" (UID: \"d30312af-f1f2-40e0-b630-db46b93837eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a7b4e06-xv9vg" Oct 30 21:39:58.633117 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:58.633091 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30312af-f1f2-40e0-b630-db46b93837eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a7b4e06-xv9vg\" (UID: \"d30312af-f1f2-40e0-b630-db46b93837eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a7b4e06-xv9vg" Oct 30 21:39:58.688904 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:58.688831 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4pk9\" (UniqueName: \"kubernetes.io/projected/d30312af-f1f2-40e0-b630-db46b93837eb-kube-api-access-d4pk9\") pod \"maas-api-revoke-sub-1a7b4e06-xv9vg\" (UID: \"d30312af-f1f2-40e0-b630-db46b93837eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a7b4e06-xv9vg" Oct 30 21:39:58.701016 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:58.700987 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a7b4e06-xv9vg" Oct 30 21:39:59.069018 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:59.068983 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a7b4e06-xv9vg"] Oct 30 21:39:59.075528 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:39:59.075501 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd30312af_f1f2_40e0_b630_db46b93837eb.slice/crio-4688af4ceb3c928b93d8789f9ce183e0d95a32dcfddfc28dfc0658a198fb2246 WatchSource:0}: Error finding container 4688af4ceb3c928b93d8789f9ce183e0d95a32dcfddfc28dfc0658a198fb2246: Status 404 returned error can't find the container with id 4688af4ceb3c928b93d8789f9ce183e0d95a32dcfddfc28dfc0658a198fb2246 Oct 30 21:39:59.297201 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:59.297170 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd"] Oct 30 21:39:59.301703 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:59.301682 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" Oct 30 21:39:59.313394 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:59.313369 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a7b4e06-xv9vg" event={"ID":"d30312af-f1f2-40e0-b630-db46b93837eb","Type":"ContainerStarted","Data":"4688af4ceb3c928b93d8789f9ce183e0d95a32dcfddfc28dfc0658a198fb2246"} Oct 30 21:39:59.350788 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:59.350762 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-554f46-openshift-default-dockercfg-lpld4\"" Oct 30 21:39:59.399171 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:59.399142 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd"] Oct 30 21:39:59.438207 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:59.438178 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-istio-podinfo\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd\" (UID: \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" Oct 30 21:39:59.438406 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:59.438218 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-istio-data\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd\" (UID: \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" Oct 30 21:39:59.438406 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:59.438244 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-istio-token\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd\" (UID: \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" Oct 30 21:39:59.438406 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:59.438383 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-credential-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd\" (UID: \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" Oct 30 21:39:59.438561 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:59.438424 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-workload-certs\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd\" (UID: \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" Oct 30 21:39:59.438561 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:59.438441 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-istiod-ca-cert\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd\" (UID: \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" Oct 30 21:39:59.438665 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:59.438562 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-istio-envoy\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd\" (UID: \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" Oct 30 21:39:59.438665 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:59.438614 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-workload-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd\" (UID: \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" Oct 30 21:39:59.438665 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:59.438640 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98wf9\" (UniqueName: \"kubernetes.io/projected/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-kube-api-access-98wf9\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd\" (UID: \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" Oct 30 21:39:59.539403 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:59.539368 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-workload-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd\" (UID: \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" Oct 30 21:39:59.539550 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:59.539420 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98wf9\" (UniqueName: \"kubernetes.io/projected/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-kube-api-access-98wf9\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd\" (UID: \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" Oct 30 21:39:59.539550 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:59.539452 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-istio-podinfo\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd\" (UID: \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" Oct 30 21:39:59.539550 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:59.539479 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-istio-data\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd\" (UID: \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" Oct 30 21:39:59.539550 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:59.539509 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-istio-token\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd\" (UID: \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" Oct 30 21:39:59.539550 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:59.539538 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-credential-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd\" (UID: \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" Oct 30 21:39:59.539830 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:59.539579 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-workload-certs\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd\" (UID: \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" Oct 30 21:39:59.539830 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:59.539601 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-istiod-ca-cert\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd\" (UID: \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" Oct 30 21:39:59.539830 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:59.539652 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-istio-envoy\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd\" (UID: \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" Oct 30 21:39:59.540221 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:59.539827 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-workload-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd\" (UID: \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" Oct 30 21:39:59.540221 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:59.540133 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-istio-data\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd\" (UID: \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" Oct 30 21:39:59.540221 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:59.540167 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-workload-certs\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd\" (UID: \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" Oct 30 21:39:59.540399 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:59.540236 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-credential-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd\" (UID: \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" Oct 30 21:39:59.540545 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:59.540518 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-istiod-ca-cert\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd\" (UID: \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" Oct 30 21:39:59.542009 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:59.541982 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-istio-envoy\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd\" (UID: \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" Oct 30 21:39:59.542563 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:59.542543 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-istio-podinfo\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd\" (UID: \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" Oct 30 21:39:59.597799 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:59.597726 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-istio-token\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd\" (UID: \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" Oct 30 21:39:59.597799 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:59.597776 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98wf9\" (UniqueName: \"kubernetes.io/projected/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-kube-api-access-98wf9\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd\" (UID: \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" Oct 30 21:39:59.614682 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:39:59.614654 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" Oct 30 21:40:00.052710 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:00.052671 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd"] Oct 30 21:40:00.065892 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:00.065853 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:40:00.065998 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:00.065930 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:40:00.065998 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:00.065959 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:40:00.320734 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:00.320638 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" event={"ID":"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b","Type":"ContainerStarted","Data":"2b0db5528a66e23021aee1e96044a644cb5fe268863b6ba26c58c267afb18bb4"} Oct 30 21:40:00.320734 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:00.320705 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" event={"ID":"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b","Type":"ContainerStarted","Data":"8358ed816053f4d4d2415495dc623126bb24660ee342b95846d48e1417008f29"} Oct 30 21:40:00.322320 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:00.322291 4560 generic.go:356] "Generic (PLEG): container finished" podID="d30312af-f1f2-40e0-b630-db46b93837eb" containerID="ceb54cb8b75ed349d77c81843fa396176328a7e98f34406e1b7911849f787fa8" exitCode=0 Oct 30 21:40:00.322473 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:00.322329 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a7b4e06-xv9vg" event={"ID":"d30312af-f1f2-40e0-b630-db46b93837eb","Type":"ContainerDied","Data":"ceb54cb8b75ed349d77c81843fa396176328a7e98f34406e1b7911849f787fa8"} Oct 30 21:40:00.435777 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:00.435721 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" podStartSLOduration=1.435701366 podStartE2EDuration="1.435701366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:00.434660112 +0000 UTC m=+1575.989359890" watchObservedRunningTime="2026-10-30 21:40:00.435701366 +0000 UTC m=+1575.990401144" Oct 30 21:40:00.484326 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:00.484290 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03a645de-426dc"] Oct 30 21:40:00.488364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:00.488328 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a645de-426dc" Oct 30 21:40:00.535607 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:00.535576 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03a645de-426dc"] Oct 30 21:40:00.615374 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:00.615282 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" Oct 30 21:40:00.653321 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:00.653280 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv4hb\" (UniqueName: \"kubernetes.io/projected/42451875-993c-4186-a4a6-fad56871700a-kube-api-access-xv4hb\") pod \"maas-api-revoke-sub-03a645de-426dc\" (UID: \"42451875-993c-4186-a4a6-fad56871700a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a645de-426dc" Oct 30 21:40:00.653510 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:00.653367 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42451875-993c-4186-a4a6-fad56871700a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03a645de-426dc\" (UID: \"42451875-993c-4186-a4a6-fad56871700a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a645de-426dc" Oct 30 21:40:00.653510 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:00.653473 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42451875-993c-4186-a4a6-fad56871700a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03a645de-426dc\" (UID: \"42451875-993c-4186-a4a6-fad56871700a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a645de-426dc" Oct 30 21:40:00.754942 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:00.754899 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv4hb\" (UniqueName: \"kubernetes.io/projected/42451875-993c-4186-a4a6-fad56871700a-kube-api-access-xv4hb\") pod \"maas-api-revoke-sub-03a645de-426dc\" (UID: \"42451875-993c-4186-a4a6-fad56871700a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a645de-426dc" Oct 30 21:40:00.755164 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:00.754968 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42451875-993c-4186-a4a6-fad56871700a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03a645de-426dc\" (UID: \"42451875-993c-4186-a4a6-fad56871700a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a645de-426dc" Oct 30 21:40:00.755164 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:00.755006 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42451875-993c-4186-a4a6-fad56871700a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03a645de-426dc\" (UID: \"42451875-993c-4186-a4a6-fad56871700a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a645de-426dc" Oct 30 21:40:00.755738 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:00.755707 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42451875-993c-4186-a4a6-fad56871700a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03a645de-426dc\" (UID: \"42451875-993c-4186-a4a6-fad56871700a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a645de-426dc" Oct 30 21:40:00.757475 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:00.757453 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42451875-993c-4186-a4a6-fad56871700a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03a645de-426dc\" (UID: \"42451875-993c-4186-a4a6-fad56871700a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a645de-426dc" Oct 30 21:40:00.812860 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:00.812831 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv4hb\" (UniqueName: \"kubernetes.io/projected/42451875-993c-4186-a4a6-fad56871700a-kube-api-access-xv4hb\") pod \"maas-api-revoke-sub-03a645de-426dc\" (UID: \"42451875-993c-4186-a4a6-fad56871700a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a645de-426dc" Oct 30 21:40:01.103849 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:01.103816 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a645de-426dc" Oct 30 21:40:01.505424 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:01.505394 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03a645de-426dc"] Oct 30 21:40:01.506742 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:01.506728 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a7b4e06-xv9vg" Oct 30 21:40:01.511610 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:40:01.511583 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42451875_993c_4186_a4a6_fad56871700a.slice/crio-c76892b0d6e8423328ec66c9b6117719b276fb69c8071fec7c8f533f03a350ca WatchSource:0}: Error finding container c76892b0d6e8423328ec66c9b6117719b276fb69c8071fec7c8f533f03a350ca: Status 404 returned error can't find the container with id c76892b0d6e8423328ec66c9b6117719b276fb69c8071fec7c8f533f03a350ca Oct 30 21:40:01.617155 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:01.617119 4560 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 21:40:01.617297 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:01.617171 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" podUID="bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 21:40:01.663478 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:01.663381 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d30312af-f1f2-40e0-b630-db46b93837eb-kube-api-access-d4pk9\" (UniqueName: \"kubernetes.io/projected/d30312af-f1f2-40e0-b630-db46b93837eb-kube-api-access-d4pk9\") pod \"d30312af-f1f2-40e0-b630-db46b93837eb\" (UID: \"d30312af-f1f2-40e0-b630-db46b93837eb\") " Oct 30 21:40:01.663478 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:01.663470 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d30312af-f1f2-40e0-b630-db46b93837eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30312af-f1f2-40e0-b630-db46b93837eb-maas-api-openshift-ca\") pod \"d30312af-f1f2-40e0-b630-db46b93837eb\" (UID: \"d30312af-f1f2-40e0-b630-db46b93837eb\") " Oct 30 21:40:01.663668 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:01.663607 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d30312af-f1f2-40e0-b630-db46b93837eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30312af-f1f2-40e0-b630-db46b93837eb-maas-api-xks-ca\") pod \"d30312af-f1f2-40e0-b630-db46b93837eb\" (UID: \"d30312af-f1f2-40e0-b630-db46b93837eb\") " Oct 30 21:40:01.664065 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:01.664038 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d30312af-f1f2-40e0-b630-db46b93837eb-maas-api-openshift-ca" pod "d30312af-f1f2-40e0-b630-db46b93837eb" (UID: "d30312af-f1f2-40e0-b630-db46b93837eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:01.665676 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:01.665648 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d30312af-f1f2-40e0-b630-db46b93837eb-kube-api-access-d4pk9" pod "d30312af-f1f2-40e0-b630-db46b93837eb" (UID: "d30312af-f1f2-40e0-b630-db46b93837eb"). InnerVolumeSpecName "kube-api-access-d4pk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:01.665774 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:01.665750 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d30312af-f1f2-40e0-b630-db46b93837eb-maas-api-xks-ca" pod "d30312af-f1f2-40e0-b630-db46b93837eb" (UID: "d30312af-f1f2-40e0-b630-db46b93837eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:01.764786 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:01.764751 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4pk9\" (UniqueName: \"kubernetes.io/projected/d30312af-f1f2-40e0-b630-db46b93837eb-kube-api-access-d4pk9\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:01.764786 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:01.764779 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30312af-f1f2-40e0-b630-db46b93837eb-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:01.764975 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:01.764792 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30312af-f1f2-40e0-b630-db46b93837eb-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:02.362027 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:02.361991 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a645de-426dc" event={"ID":"42451875-993c-4186-a4a6-fad56871700a","Type":"ContainerStarted","Data":"e0c4e8b8457c9b5a77b79bd2e5b49fafae11ec178d608c33346c84e4a3e2ce45"} Oct 30 21:40:02.362219 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:02.362054 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a645de-426dc" event={"ID":"42451875-993c-4186-a4a6-fad56871700a","Type":"ContainerStarted","Data":"c76892b0d6e8423328ec66c9b6117719b276fb69c8071fec7c8f533f03a350ca"} Oct 30 21:40:02.364384 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:02.364362 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a7b4e06-xv9vg" Oct 30 21:40:02.364384 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:02.364366 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a7b4e06-xv9vg" event={"ID":"d30312af-f1f2-40e0-b630-db46b93837eb","Type":"ContainerDied","Data":"4688af4ceb3c928b93d8789f9ce183e0d95a32dcfddfc28dfc0658a198fb2246"} Oct 30 21:40:02.364577 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:02.364413 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4688af4ceb3c928b93d8789f9ce183e0d95a32dcfddfc28dfc0658a198fb2246" Oct 30 21:40:02.474514 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:02.474453 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a645de-426dc" podStartSLOduration=2.474438297 podStartE2EDuration="2.474438297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:02.472722852 +0000 UTC m=+1578.027422630" watchObservedRunningTime="2026-10-30 21:40:02.474438297 +0000 UTC m=+1578.029138073" Oct 30 21:40:02.615262 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:02.615179 4560 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 21:40:02.615262 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:02.615231 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" podUID="bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 21:40:03.370844 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:03.370811 4560 generic.go:356] "Generic (PLEG): container finished" podID="42451875-993c-4186-a4a6-fad56871700a" containerID="e0c4e8b8457c9b5a77b79bd2e5b49fafae11ec178d608c33346c84e4a3e2ce45" exitCode=0 Oct 30 21:40:03.370999 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:03.370856 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a645de-426dc" event={"ID":"42451875-993c-4186-a4a6-fad56871700a","Type":"ContainerDied","Data":"e0c4e8b8457c9b5a77b79bd2e5b49fafae11ec178d608c33346c84e4a3e2ce45"} Oct 30 21:40:03.615501 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:03.615466 4560 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 21:40:03.615894 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:03.615519 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" podUID="bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 21:40:04.557571 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:04.557542 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a645de-426dc" Oct 30 21:40:04.615897 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:04.615866 4560 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 21:40:04.616332 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:04.615929 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" podUID="bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 21:40:04.708720 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:04.708644 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42451875-993c-4186-a4a6-fad56871700a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42451875-993c-4186-a4a6-fad56871700a-maas-api-openshift-ca\") pod \"42451875-993c-4186-a4a6-fad56871700a\" (UID: \"42451875-993c-4186-a4a6-fad56871700a\") " Oct 30 21:40:04.708720 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:04.708682 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42451875-993c-4186-a4a6-fad56871700a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42451875-993c-4186-a4a6-fad56871700a-maas-api-xks-ca\") pod \"42451875-993c-4186-a4a6-fad56871700a\" (UID: \"42451875-993c-4186-a4a6-fad56871700a\") " Oct 30 21:40:04.708928 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:04.708766 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42451875-993c-4186-a4a6-fad56871700a-kube-api-access-xv4hb\" (UniqueName: \"kubernetes.io/projected/42451875-993c-4186-a4a6-fad56871700a-kube-api-access-xv4hb\") pod \"42451875-993c-4186-a4a6-fad56871700a\" (UID: \"42451875-993c-4186-a4a6-fad56871700a\") " Oct 30 21:40:04.709069 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:04.709040 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42451875-993c-4186-a4a6-fad56871700a-maas-api-openshift-ca" pod "42451875-993c-4186-a4a6-fad56871700a" (UID: "42451875-993c-4186-a4a6-fad56871700a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:04.710968 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:04.710933 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42451875-993c-4186-a4a6-fad56871700a-kube-api-access-xv4hb" pod "42451875-993c-4186-a4a6-fad56871700a" (UID: "42451875-993c-4186-a4a6-fad56871700a"). InnerVolumeSpecName "kube-api-access-xv4hb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:04.711082 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:04.710980 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42451875-993c-4186-a4a6-fad56871700a-maas-api-xks-ca" pod "42451875-993c-4186-a4a6-fad56871700a" (UID: "42451875-993c-4186-a4a6-fad56871700a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:04.810480 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:04.810439 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42451875-993c-4186-a4a6-fad56871700a-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:04.810480 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:04.810472 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42451875-993c-4186-a4a6-fad56871700a-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:04.810480 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:04.810485 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv4hb\" (UniqueName: \"kubernetes.io/projected/42451875-993c-4186-a4a6-fad56871700a-kube-api-access-xv4hb\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:05.382751 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:05.382721 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a645de-426dc" event={"ID":"42451875-993c-4186-a4a6-fad56871700a","Type":"ContainerDied","Data":"c76892b0d6e8423328ec66c9b6117719b276fb69c8071fec7c8f533f03a350ca"} Oct 30 21:40:05.382751 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:05.382740 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a645de-426dc" Oct 30 21:40:05.382951 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:05.382773 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c76892b0d6e8423328ec66c9b6117719b276fb69c8071fec7c8f533f03a350ca" Oct 30 21:40:05.615701 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:05.615662 4560 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 21:40:05.615862 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:05.615719 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" podUID="bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 21:40:06.762933 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:06.762895 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" Oct 30 21:40:06.763925 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:06.763796 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" Oct 30 21:40:06.764919 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:06.764898 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" Oct 30 21:40:07.299440 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:07.299407 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b05894f-ktjr2"] Oct 30 21:40:07.299782 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:07.299770 4560 cpu_manager.go:414] "containerMap: removing container" podUID="d30312af-f1f2-40e0-b630-db46b93837eb" containerName="revoke-keys" Oct 30 21:40:07.299830 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:07.299783 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d30312af-f1f2-40e0-b630-db46b93837eb" containerName="revoke-keys" Oct 30 21:40:07.299830 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:07.299790 4560 cpu_manager.go:414] "containerMap: removing container" podUID="42451875-993c-4186-a4a6-fad56871700a" containerName="revoke-keys" Oct 30 21:40:07.299830 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:07.299795 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42451875-993c-4186-a4a6-fad56871700a" containerName="revoke-keys" Oct 30 21:40:07.299942 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:07.299851 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="d30312af-f1f2-40e0-b630-db46b93837eb" containerName="revoke-keys" Oct 30 21:40:07.299942 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:07.299859 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="42451875-993c-4186-a4a6-fad56871700a" containerName="revoke-keys" Oct 30 21:40:07.302656 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:07.302638 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b05894f-ktjr2" Oct 30 21:40:07.352794 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:07.352763 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:40:07.352794 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:07.352774 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:40:07.352998 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:07.352797 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:40:07.352998 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:07.352798 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:40:07.360815 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:07.360786 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b05894f-ktjr2"] Oct 30 21:40:07.436756 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:07.436728 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq8ns\" (UniqueName: \"kubernetes.io/projected/1330052e-81d2-474e-ace8-c98d0f8587c6-kube-api-access-fq8ns\") pod \"maas-api-revoke-sub-3b05894f-ktjr2\" (UID: \"1330052e-81d2-474e-ace8-c98d0f8587c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b05894f-ktjr2" Oct 30 21:40:07.436922 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:07.436847 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1330052e-81d2-474e-ace8-c98d0f8587c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b05894f-ktjr2\" (UID: \"1330052e-81d2-474e-ace8-c98d0f8587c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b05894f-ktjr2" Oct 30 21:40:07.436922 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:07.436899 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1330052e-81d2-474e-ace8-c98d0f8587c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b05894f-ktjr2\" (UID: \"1330052e-81d2-474e-ace8-c98d0f8587c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b05894f-ktjr2" Oct 30 21:40:07.513604 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:07.513571 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-751f022a-f2hvf"] Oct 30 21:40:07.517701 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:07.517677 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-751f022a-f2hvf" Oct 30 21:40:07.538195 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:07.538169 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq8ns\" (UniqueName: \"kubernetes.io/projected/1330052e-81d2-474e-ace8-c98d0f8587c6-kube-api-access-fq8ns\") pod \"maas-api-revoke-sub-3b05894f-ktjr2\" (UID: \"1330052e-81d2-474e-ace8-c98d0f8587c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b05894f-ktjr2" Oct 30 21:40:07.538372 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:07.538273 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1330052e-81d2-474e-ace8-c98d0f8587c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b05894f-ktjr2\" (UID: \"1330052e-81d2-474e-ace8-c98d0f8587c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b05894f-ktjr2" Oct 30 21:40:07.538372 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:07.538310 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1330052e-81d2-474e-ace8-c98d0f8587c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b05894f-ktjr2\" (UID: \"1330052e-81d2-474e-ace8-c98d0f8587c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b05894f-ktjr2" Oct 30 21:40:07.539007 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:07.538985 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1330052e-81d2-474e-ace8-c98d0f8587c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b05894f-ktjr2\" (UID: \"1330052e-81d2-474e-ace8-c98d0f8587c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b05894f-ktjr2" Oct 30 21:40:07.540673 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:07.540648 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1330052e-81d2-474e-ace8-c98d0f8587c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b05894f-ktjr2\" (UID: \"1330052e-81d2-474e-ace8-c98d0f8587c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b05894f-ktjr2" Oct 30 21:40:07.574069 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:07.574019 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-751f022a-f2hvf"] Oct 30 21:40:07.599976 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:07.599946 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq8ns\" (UniqueName: \"kubernetes.io/projected/1330052e-81d2-474e-ace8-c98d0f8587c6-kube-api-access-fq8ns\") pod \"maas-api-revoke-sub-3b05894f-ktjr2\" (UID: \"1330052e-81d2-474e-ace8-c98d0f8587c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b05894f-ktjr2" Oct 30 21:40:07.615523 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:07.615503 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b05894f-ktjr2" Oct 30 21:40:07.639697 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:07.639675 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f044066-13f0-4162-8aae-74ec3062e041-maas-api-xks-ca\") pod \"maas-api-revoke-sub-751f022a-f2hvf\" (UID: \"8f044066-13f0-4162-8aae-74ec3062e041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-751f022a-f2hvf" Oct 30 21:40:07.639795 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:07.639720 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp694\" (UniqueName: \"kubernetes.io/projected/8f044066-13f0-4162-8aae-74ec3062e041-kube-api-access-bp694\") pod \"maas-api-revoke-sub-751f022a-f2hvf\" (UID: \"8f044066-13f0-4162-8aae-74ec3062e041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-751f022a-f2hvf" Oct 30 21:40:07.639856 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:07.639838 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f044066-13f0-4162-8aae-74ec3062e041-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-751f022a-f2hvf\" (UID: \"8f044066-13f0-4162-8aae-74ec3062e041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-751f022a-f2hvf" Oct 30 21:40:07.740863 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:07.740830 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f044066-13f0-4162-8aae-74ec3062e041-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-751f022a-f2hvf\" (UID: \"8f044066-13f0-4162-8aae-74ec3062e041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-751f022a-f2hvf" Oct 30 21:40:07.741027 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:07.740899 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f044066-13f0-4162-8aae-74ec3062e041-maas-api-xks-ca\") pod \"maas-api-revoke-sub-751f022a-f2hvf\" (UID: \"8f044066-13f0-4162-8aae-74ec3062e041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-751f022a-f2hvf" Oct 30 21:40:07.741027 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:07.740927 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp694\" (UniqueName: \"kubernetes.io/projected/8f044066-13f0-4162-8aae-74ec3062e041-kube-api-access-bp694\") pod \"maas-api-revoke-sub-751f022a-f2hvf\" (UID: \"8f044066-13f0-4162-8aae-74ec3062e041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-751f022a-f2hvf" Oct 30 21:40:07.741584 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:07.741559 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f044066-13f0-4162-8aae-74ec3062e041-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-751f022a-f2hvf\" (UID: \"8f044066-13f0-4162-8aae-74ec3062e041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-751f022a-f2hvf" Oct 30 21:40:07.743391 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:07.743371 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f044066-13f0-4162-8aae-74ec3062e041-maas-api-xks-ca\") pod \"maas-api-revoke-sub-751f022a-f2hvf\" (UID: \"8f044066-13f0-4162-8aae-74ec3062e041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-751f022a-f2hvf" Oct 30 21:40:07.800906 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:07.800883 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp694\" (UniqueName: \"kubernetes.io/projected/8f044066-13f0-4162-8aae-74ec3062e041-kube-api-access-bp694\") pod \"maas-api-revoke-sub-751f022a-f2hvf\" (UID: \"8f044066-13f0-4162-8aae-74ec3062e041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-751f022a-f2hvf" Oct 30 21:40:07.829756 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:07.829706 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-751f022a-f2hvf" Oct 30 21:40:08.001977 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:08.001943 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b05894f-ktjr2"] Oct 30 21:40:08.008204 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:40:08.008169 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1330052e_81d2_474e_ace8_c98d0f8587c6.slice/crio-04785226b6b6ecae5f2b9c17661c661b713424c096f0db9d4245b25b33bc27a6 WatchSource:0}: Error finding container 04785226b6b6ecae5f2b9c17661c661b713424c096f0db9d4245b25b33bc27a6: Status 404 returned error can't find the container with id 04785226b6b6ecae5f2b9c17661c661b713424c096f0db9d4245b25b33bc27a6 Oct 30 21:40:08.213778 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:08.213703 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-751f022a-f2hvf"] Oct 30 21:40:08.221401 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:40:08.221374 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f044066_13f0_4162_8aae_74ec3062e041.slice/crio-307e3657efdebb8a43cadbf22287e84b4b1794eccb2ad12ada508857a6ed6a68 WatchSource:0}: Error finding container 307e3657efdebb8a43cadbf22287e84b4b1794eccb2ad12ada508857a6ed6a68: Status 404 returned error can't find the container with id 307e3657efdebb8a43cadbf22287e84b4b1794eccb2ad12ada508857a6ed6a68 Oct 30 21:40:08.399740 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:08.399705 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b05894f-ktjr2" event={"ID":"1330052e-81d2-474e-ace8-c98d0f8587c6","Type":"ContainerStarted","Data":"752ae82acd8176b28cb4ad563c799fc7f198866aae51cd3aee70028895bc4c41"} Oct 30 21:40:08.399915 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:08.399764 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b05894f-ktjr2" event={"ID":"1330052e-81d2-474e-ace8-c98d0f8587c6","Type":"ContainerStarted","Data":"04785226b6b6ecae5f2b9c17661c661b713424c096f0db9d4245b25b33bc27a6"} Oct 30 21:40:08.401016 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:08.400986 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-751f022a-f2hvf" event={"ID":"8f044066-13f0-4162-8aae-74ec3062e041","Type":"ContainerStarted","Data":"307e3657efdebb8a43cadbf22287e84b4b1794eccb2ad12ada508857a6ed6a68"} Oct 30 21:40:08.516280 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:08.516221 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b05894f-ktjr2" podStartSLOduration=1.516202965 podStartE2EDuration="1.516202965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:08.515131039 +0000 UTC m=+1584.069830816" watchObservedRunningTime="2026-10-30 21:40:08.516202965 +0000 UTC m=+1584.070902744" Oct 30 21:40:09.407582 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:09.407547 4560 generic.go:356] "Generic (PLEG): container finished" podID="1330052e-81d2-474e-ace8-c98d0f8587c6" containerID="752ae82acd8176b28cb4ad563c799fc7f198866aae51cd3aee70028895bc4c41" exitCode=0 Oct 30 21:40:09.408034 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:09.407638 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b05894f-ktjr2" event={"ID":"1330052e-81d2-474e-ace8-c98d0f8587c6","Type":"ContainerDied","Data":"752ae82acd8176b28cb4ad563c799fc7f198866aae51cd3aee70028895bc4c41"} Oct 30 21:40:09.409162 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:09.409141 4560 generic.go:356] "Generic (PLEG): container finished" podID="8f044066-13f0-4162-8aae-74ec3062e041" containerID="d28b7a41851f31dc1d3f160f1027bffe50ed095b1b670c8eab86d4caa917e0ce" exitCode=0 Oct 30 21:40:09.409272 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:09.409195 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-751f022a-f2hvf" event={"ID":"8f044066-13f0-4162-8aae-74ec3062e041","Type":"ContainerDied","Data":"d28b7a41851f31dc1d3f160f1027bffe50ed095b1b670c8eab86d4caa917e0ce"} Oct 30 21:40:10.551727 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:10.551700 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-751f022a-f2hvf" Oct 30 21:40:10.561940 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:10.561919 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f044066-13f0-4162-8aae-74ec3062e041-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f044066-13f0-4162-8aae-74ec3062e041-maas-api-xks-ca\") pod \"8f044066-13f0-4162-8aae-74ec3062e041\" (UID: \"8f044066-13f0-4162-8aae-74ec3062e041\") " Oct 30 21:40:10.562056 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:10.562017 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f044066-13f0-4162-8aae-74ec3062e041-kube-api-access-bp694\" (UniqueName: \"kubernetes.io/projected/8f044066-13f0-4162-8aae-74ec3062e041-kube-api-access-bp694\") pod \"8f044066-13f0-4162-8aae-74ec3062e041\" (UID: \"8f044066-13f0-4162-8aae-74ec3062e041\") " Oct 30 21:40:10.562123 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:10.562066 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f044066-13f0-4162-8aae-74ec3062e041-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f044066-13f0-4162-8aae-74ec3062e041-maas-api-openshift-ca\") pod \"8f044066-13f0-4162-8aae-74ec3062e041\" (UID: \"8f044066-13f0-4162-8aae-74ec3062e041\") " Oct 30 21:40:10.562506 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:10.562477 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f044066-13f0-4162-8aae-74ec3062e041-maas-api-openshift-ca" pod "8f044066-13f0-4162-8aae-74ec3062e041" (UID: "8f044066-13f0-4162-8aae-74ec3062e041"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:10.564005 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:10.563983 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f044066-13f0-4162-8aae-74ec3062e041-kube-api-access-bp694" pod "8f044066-13f0-4162-8aae-74ec3062e041" (UID: "8f044066-13f0-4162-8aae-74ec3062e041"). InnerVolumeSpecName "kube-api-access-bp694". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:10.564370 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:10.564350 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f044066-13f0-4162-8aae-74ec3062e041-maas-api-xks-ca" pod "8f044066-13f0-4162-8aae-74ec3062e041" (UID: "8f044066-13f0-4162-8aae-74ec3062e041"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:10.581434 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:10.581414 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b05894f-ktjr2" Oct 30 21:40:10.663125 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:10.663098 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1330052e-81d2-474e-ace8-c98d0f8587c6-kube-api-access-fq8ns\" (UniqueName: \"kubernetes.io/projected/1330052e-81d2-474e-ace8-c98d0f8587c6-kube-api-access-fq8ns\") pod \"1330052e-81d2-474e-ace8-c98d0f8587c6\" (UID: \"1330052e-81d2-474e-ace8-c98d0f8587c6\") " Oct 30 21:40:10.663231 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:10.663196 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1330052e-81d2-474e-ace8-c98d0f8587c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1330052e-81d2-474e-ace8-c98d0f8587c6-maas-api-openshift-ca\") pod \"1330052e-81d2-474e-ace8-c98d0f8587c6\" (UID: \"1330052e-81d2-474e-ace8-c98d0f8587c6\") " Oct 30 21:40:10.663330 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:10.663301 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1330052e-81d2-474e-ace8-c98d0f8587c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1330052e-81d2-474e-ace8-c98d0f8587c6-maas-api-xks-ca\") pod \"1330052e-81d2-474e-ace8-c98d0f8587c6\" (UID: \"1330052e-81d2-474e-ace8-c98d0f8587c6\") " Oct 30 21:40:10.663575 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:10.663544 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1330052e-81d2-474e-ace8-c98d0f8587c6-maas-api-openshift-ca" pod "1330052e-81d2-474e-ace8-c98d0f8587c6" (UID: "1330052e-81d2-474e-ace8-c98d0f8587c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:10.663742 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:10.663723 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp694\" (UniqueName: \"kubernetes.io/projected/8f044066-13f0-4162-8aae-74ec3062e041-kube-api-access-bp694\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:10.663857 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:10.663747 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f044066-13f0-4162-8aae-74ec3062e041-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:10.663857 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:10.663762 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1330052e-81d2-474e-ace8-c98d0f8587c6-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:10.663857 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:10.663780 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f044066-13f0-4162-8aae-74ec3062e041-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:10.665729 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:10.665693 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1330052e-81d2-474e-ace8-c98d0f8587c6-kube-api-access-fq8ns" pod "1330052e-81d2-474e-ace8-c98d0f8587c6" (UID: "1330052e-81d2-474e-ace8-c98d0f8587c6"). InnerVolumeSpecName "kube-api-access-fq8ns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:10.665829 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:10.665739 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1330052e-81d2-474e-ace8-c98d0f8587c6-maas-api-xks-ca" pod "1330052e-81d2-474e-ace8-c98d0f8587c6" (UID: "1330052e-81d2-474e-ace8-c98d0f8587c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:10.765152 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:10.765124 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq8ns\" (UniqueName: \"kubernetes.io/projected/1330052e-81d2-474e-ace8-c98d0f8587c6-kube-api-access-fq8ns\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:10.765152 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:10.765148 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1330052e-81d2-474e-ace8-c98d0f8587c6-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:11.420325 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:11.420287 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b05894f-ktjr2" Oct 30 21:40:11.420325 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:11.420306 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b05894f-ktjr2" event={"ID":"1330052e-81d2-474e-ace8-c98d0f8587c6","Type":"ContainerDied","Data":"04785226b6b6ecae5f2b9c17661c661b713424c096f0db9d4245b25b33bc27a6"} Oct 30 21:40:11.420586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:11.420381 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04785226b6b6ecae5f2b9c17661c661b713424c096f0db9d4245b25b33bc27a6" Oct 30 21:40:11.443136 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:11.421953 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-751f022a-f2hvf" event={"ID":"8f044066-13f0-4162-8aae-74ec3062e041","Type":"ContainerDied","Data":"307e3657efdebb8a43cadbf22287e84b4b1794eccb2ad12ada508857a6ed6a68"} Oct 30 21:40:11.443136 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:11.421996 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="307e3657efdebb8a43cadbf22287e84b4b1794eccb2ad12ada508857a6ed6a68" Oct 30 21:40:11.443136 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:11.422054 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-751f022a-f2hvf" Oct 30 21:40:19.687007 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:40:19.686969 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:40:20.236505 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:20.236466 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aec7d28-wl4wq"] Oct 30 21:40:20.236900 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:20.236887 4560 cpu_manager.go:414] "containerMap: removing container" podUID="8f044066-13f0-4162-8aae-74ec3062e041" containerName="revoke-keys" Oct 30 21:40:20.236954 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:20.236903 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f044066-13f0-4162-8aae-74ec3062e041" containerName="revoke-keys" Oct 30 21:40:20.236954 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:20.236914 4560 cpu_manager.go:414] "containerMap: removing container" podUID="1330052e-81d2-474e-ace8-c98d0f8587c6" containerName="revoke-keys" Oct 30 21:40:20.236954 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:20.236920 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1330052e-81d2-474e-ace8-c98d0f8587c6" containerName="revoke-keys" Oct 30 21:40:20.237046 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:20.236980 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f044066-13f0-4162-8aae-74ec3062e041" containerName="revoke-keys" Oct 30 21:40:20.237046 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:20.236989 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="1330052e-81d2-474e-ace8-c98d0f8587c6" containerName="revoke-keys" Oct 30 21:40:20.240452 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:20.240431 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aec7d28-wl4wq" Oct 30 21:40:20.291986 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:20.291955 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:40:20.338416 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:20.338372 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:40:20.338416 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:20.338404 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:40:20.338619 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:20.338443 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:40:20.340687 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:20.340667 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aec7d28-wl4wq"] Oct 30 21:40:20.357301 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:20.357272 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgk5k\" (UniqueName: \"kubernetes.io/projected/6a7da446-6981-4258-aca5-affa0cc7bb41-kube-api-access-lgk5k\") pod \"maas-api-revoke-sub-5aec7d28-wl4wq\" (UID: \"6a7da446-6981-4258-aca5-affa0cc7bb41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aec7d28-wl4wq" Oct 30 21:40:20.357439 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:20.357417 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a7da446-6981-4258-aca5-affa0cc7bb41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5aec7d28-wl4wq\" (UID: \"6a7da446-6981-4258-aca5-affa0cc7bb41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aec7d28-wl4wq" Oct 30 21:40:20.357480 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:20.357450 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a7da446-6981-4258-aca5-affa0cc7bb41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5aec7d28-wl4wq\" (UID: \"6a7da446-6981-4258-aca5-affa0cc7bb41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aec7d28-wl4wq" Oct 30 21:40:20.459032 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:20.458997 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a7da446-6981-4258-aca5-affa0cc7bb41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5aec7d28-wl4wq\" (UID: \"6a7da446-6981-4258-aca5-affa0cc7bb41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aec7d28-wl4wq" Oct 30 21:40:20.459032 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:20.459036 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a7da446-6981-4258-aca5-affa0cc7bb41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5aec7d28-wl4wq\" (UID: \"6a7da446-6981-4258-aca5-affa0cc7bb41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aec7d28-wl4wq" Oct 30 21:40:20.459281 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:20.459089 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgk5k\" (UniqueName: \"kubernetes.io/projected/6a7da446-6981-4258-aca5-affa0cc7bb41-kube-api-access-lgk5k\") pod \"maas-api-revoke-sub-5aec7d28-wl4wq\" (UID: \"6a7da446-6981-4258-aca5-affa0cc7bb41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aec7d28-wl4wq" Oct 30 21:40:20.459833 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:20.459807 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a7da446-6981-4258-aca5-affa0cc7bb41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5aec7d28-wl4wq\" (UID: \"6a7da446-6981-4258-aca5-affa0cc7bb41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aec7d28-wl4wq" Oct 30 21:40:20.461485 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:20.461463 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a7da446-6981-4258-aca5-affa0cc7bb41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5aec7d28-wl4wq\" (UID: \"6a7da446-6981-4258-aca5-affa0cc7bb41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aec7d28-wl4wq" Oct 30 21:40:20.517091 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:20.517016 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgk5k\" (UniqueName: \"kubernetes.io/projected/6a7da446-6981-4258-aca5-affa0cc7bb41-kube-api-access-lgk5k\") pod \"maas-api-revoke-sub-5aec7d28-wl4wq\" (UID: \"6a7da446-6981-4258-aca5-affa0cc7bb41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aec7d28-wl4wq" Oct 30 21:40:20.552851 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:20.552817 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aec7d28-wl4wq" Oct 30 21:40:20.925123 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:20.925039 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aec7d28-wl4wq"] Oct 30 21:40:20.931418 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:40:20.931377 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a7da446_6981_4258_aca5_affa0cc7bb41.slice/crio-380aade7466ec5a7c450a54830e1776cb8f8414ee0883b91bdd967f97b1d3b19 WatchSource:0}: Error finding container 380aade7466ec5a7c450a54830e1776cb8f8414ee0883b91bdd967f97b1d3b19: Status 404 returned error can't find the container with id 380aade7466ec5a7c450a54830e1776cb8f8414ee0883b91bdd967f97b1d3b19 Oct 30 21:40:21.472889 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:21.472853 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aec7d28-wl4wq" event={"ID":"6a7da446-6981-4258-aca5-affa0cc7bb41","Type":"ContainerStarted","Data":"ed8a7b55cb438a06a14a71b785840faeb57ed319dc362d3db65b521a621be380"} Oct 30 21:40:21.473087 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:21.472916 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aec7d28-wl4wq" event={"ID":"6a7da446-6981-4258-aca5-affa0cc7bb41","Type":"ContainerStarted","Data":"380aade7466ec5a7c450a54830e1776cb8f8414ee0883b91bdd967f97b1d3b19"} Oct 30 21:40:21.583292 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:21.583185 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aec7d28-wl4wq" podStartSLOduration=1.583168854 podStartE2EDuration="1.583168854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:21.581486087 +0000 UTC m=+1597.136185864" watchObservedRunningTime="2026-10-30 21:40:21.583168854 +0000 UTC m=+1597.137868661" Oct 30 21:40:22.478791 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:22.478759 4560 generic.go:356] "Generic (PLEG): container finished" podID="6a7da446-6981-4258-aca5-affa0cc7bb41" containerID="ed8a7b55cb438a06a14a71b785840faeb57ed319dc362d3db65b521a621be380" exitCode=0 Oct 30 21:40:22.479203 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:22.478845 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aec7d28-wl4wq" event={"ID":"6a7da446-6981-4258-aca5-affa0cc7bb41","Type":"ContainerDied","Data":"ed8a7b55cb438a06a14a71b785840faeb57ed319dc362d3db65b521a621be380"} Oct 30 21:40:23.623581 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:23.623554 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aec7d28-wl4wq" Oct 30 21:40:23.692971 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:23.692944 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a7da446-6981-4258-aca5-affa0cc7bb41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a7da446-6981-4258-aca5-affa0cc7bb41-maas-api-openshift-ca\") pod \"6a7da446-6981-4258-aca5-affa0cc7bb41\" (UID: \"6a7da446-6981-4258-aca5-affa0cc7bb41\") " Oct 30 21:40:23.693125 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:23.693072 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a7da446-6981-4258-aca5-affa0cc7bb41-kube-api-access-lgk5k\" (UniqueName: \"kubernetes.io/projected/6a7da446-6981-4258-aca5-affa0cc7bb41-kube-api-access-lgk5k\") pod \"6a7da446-6981-4258-aca5-affa0cc7bb41\" (UID: \"6a7da446-6981-4258-aca5-affa0cc7bb41\") " Oct 30 21:40:23.693125 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:23.693091 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a7da446-6981-4258-aca5-affa0cc7bb41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a7da446-6981-4258-aca5-affa0cc7bb41-maas-api-xks-ca\") pod \"6a7da446-6981-4258-aca5-affa0cc7bb41\" (UID: \"6a7da446-6981-4258-aca5-affa0cc7bb41\") " Oct 30 21:40:23.693326 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:23.693300 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a7da446-6981-4258-aca5-affa0cc7bb41-maas-api-openshift-ca" pod "6a7da446-6981-4258-aca5-affa0cc7bb41" (UID: "6a7da446-6981-4258-aca5-affa0cc7bb41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:23.693443 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:23.693429 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a7da446-6981-4258-aca5-affa0cc7bb41-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:23.695142 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:23.695119 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a7da446-6981-4258-aca5-affa0cc7bb41-maas-api-xks-ca" pod "6a7da446-6981-4258-aca5-affa0cc7bb41" (UID: "6a7da446-6981-4258-aca5-affa0cc7bb41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:23.695142 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:23.695131 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a7da446-6981-4258-aca5-affa0cc7bb41-kube-api-access-lgk5k" pod "6a7da446-6981-4258-aca5-affa0cc7bb41" (UID: "6a7da446-6981-4258-aca5-affa0cc7bb41"). InnerVolumeSpecName "kube-api-access-lgk5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:23.789860 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:23.789775 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4327b99b-l5szt"] Oct 30 21:40:23.790365 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:23.790326 4560 cpu_manager.go:414] "containerMap: removing container" podUID="6a7da446-6981-4258-aca5-affa0cc7bb41" containerName="revoke-keys" Oct 30 21:40:23.790510 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:23.790366 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a7da446-6981-4258-aca5-affa0cc7bb41" containerName="revoke-keys" Oct 30 21:40:23.790558 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:23.790511 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a7da446-6981-4258-aca5-affa0cc7bb41" containerName="revoke-keys" Oct 30 21:40:23.794384 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:23.794364 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4327b99b-l5szt" Oct 30 21:40:23.794648 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:23.794628 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a7da446-6981-4258-aca5-affa0cc7bb41-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:23.794703 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:23.794654 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgk5k\" (UniqueName: \"kubernetes.io/projected/6a7da446-6981-4258-aca5-affa0cc7bb41-kube-api-access-lgk5k\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:23.853093 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:23.853039 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4327b99b-l5szt"] Oct 30 21:40:23.895398 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:23.895334 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49da1004-a76f-46f5-925d-a6a17be6d6e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4327b99b-l5szt\" (UID: \"49da1004-a76f-46f5-925d-a6a17be6d6e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4327b99b-l5szt" Oct 30 21:40:23.895570 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:23.895496 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49da1004-a76f-46f5-925d-a6a17be6d6e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4327b99b-l5szt\" (UID: \"49da1004-a76f-46f5-925d-a6a17be6d6e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4327b99b-l5szt" Oct 30 21:40:23.895608 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:23.895568 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9nqf\" (UniqueName: \"kubernetes.io/projected/49da1004-a76f-46f5-925d-a6a17be6d6e7-kube-api-access-b9nqf\") pod \"maas-api-revoke-sub-4327b99b-l5szt\" (UID: \"49da1004-a76f-46f5-925d-a6a17be6d6e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4327b99b-l5szt" Oct 30 21:40:23.997070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:23.997027 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49da1004-a76f-46f5-925d-a6a17be6d6e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4327b99b-l5szt\" (UID: \"49da1004-a76f-46f5-925d-a6a17be6d6e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4327b99b-l5szt" Oct 30 21:40:23.997257 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:23.997113 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9nqf\" (UniqueName: \"kubernetes.io/projected/49da1004-a76f-46f5-925d-a6a17be6d6e7-kube-api-access-b9nqf\") pod \"maas-api-revoke-sub-4327b99b-l5szt\" (UID: \"49da1004-a76f-46f5-925d-a6a17be6d6e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4327b99b-l5szt" Oct 30 21:40:23.997257 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:23.997156 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49da1004-a76f-46f5-925d-a6a17be6d6e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4327b99b-l5szt\" (UID: \"49da1004-a76f-46f5-925d-a6a17be6d6e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4327b99b-l5szt" Oct 30 21:40:23.997904 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:23.997879 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49da1004-a76f-46f5-925d-a6a17be6d6e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4327b99b-l5szt\" (UID: \"49da1004-a76f-46f5-925d-a6a17be6d6e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4327b99b-l5szt" Oct 30 21:40:23.999417 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:23.999391 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49da1004-a76f-46f5-925d-a6a17be6d6e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4327b99b-l5szt\" (UID: \"49da1004-a76f-46f5-925d-a6a17be6d6e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4327b99b-l5szt" Oct 30 21:40:24.055253 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:24.055177 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9nqf\" (UniqueName: \"kubernetes.io/projected/49da1004-a76f-46f5-925d-a6a17be6d6e7-kube-api-access-b9nqf\") pod \"maas-api-revoke-sub-4327b99b-l5szt\" (UID: \"49da1004-a76f-46f5-925d-a6a17be6d6e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4327b99b-l5szt" Oct 30 21:40:24.107275 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:24.107237 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4327b99b-l5szt" Oct 30 21:40:24.356682 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:24.356607 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-329ab292-q5xcq"] Oct 30 21:40:24.361108 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:24.361040 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-329ab292-q5xcq" Oct 30 21:40:24.401142 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:24.401107 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c851243-770c-4819-abb6-b19588f0fffe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-329ab292-q5xcq\" (UID: \"0c851243-770c-4819-abb6-b19588f0fffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-329ab292-q5xcq" Oct 30 21:40:24.401282 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:24.401181 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c851243-770c-4819-abb6-b19588f0fffe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-329ab292-q5xcq\" (UID: \"0c851243-770c-4819-abb6-b19588f0fffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-329ab292-q5xcq" Oct 30 21:40:24.401282 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:24.401255 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm78d\" (UniqueName: \"kubernetes.io/projected/0c851243-770c-4819-abb6-b19588f0fffe-kube-api-access-vm78d\") pod \"maas-api-revoke-sub-329ab292-q5xcq\" (UID: \"0c851243-770c-4819-abb6-b19588f0fffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-329ab292-q5xcq" Oct 30 21:40:24.415554 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:24.415520 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-329ab292-q5xcq"] Oct 30 21:40:24.489682 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:24.489654 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aec7d28-wl4wq" Oct 30 21:40:24.489897 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:24.489693 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aec7d28-wl4wq" event={"ID":"6a7da446-6981-4258-aca5-affa0cc7bb41","Type":"ContainerDied","Data":"380aade7466ec5a7c450a54830e1776cb8f8414ee0883b91bdd967f97b1d3b19"} Oct 30 21:40:24.489897 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:24.489735 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="380aade7466ec5a7c450a54830e1776cb8f8414ee0883b91bdd967f97b1d3b19" Oct 30 21:40:24.502351 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:24.502310 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c851243-770c-4819-abb6-b19588f0fffe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-329ab292-q5xcq\" (UID: \"0c851243-770c-4819-abb6-b19588f0fffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-329ab292-q5xcq" Oct 30 21:40:24.502506 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:24.502395 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c851243-770c-4819-abb6-b19588f0fffe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-329ab292-q5xcq\" (UID: \"0c851243-770c-4819-abb6-b19588f0fffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-329ab292-q5xcq" Oct 30 21:40:24.502506 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:24.502449 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm78d\" (UniqueName: \"kubernetes.io/projected/0c851243-770c-4819-abb6-b19588f0fffe-kube-api-access-vm78d\") pod \"maas-api-revoke-sub-329ab292-q5xcq\" (UID: \"0c851243-770c-4819-abb6-b19588f0fffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-329ab292-q5xcq" Oct 30 21:40:24.503233 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:24.503203 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c851243-770c-4819-abb6-b19588f0fffe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-329ab292-q5xcq\" (UID: \"0c851243-770c-4819-abb6-b19588f0fffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-329ab292-q5xcq" Oct 30 21:40:24.504743 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:24.504723 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c851243-770c-4819-abb6-b19588f0fffe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-329ab292-q5xcq\" (UID: \"0c851243-770c-4819-abb6-b19588f0fffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-329ab292-q5xcq" Oct 30 21:40:24.529810 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:24.529778 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4327b99b-l5szt"] Oct 30 21:40:24.535712 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:40:24.535685 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49da1004_a76f_46f5_925d_a6a17be6d6e7.slice/crio-640ec633e9a874f27341ffef613b09a6535a4018175e78e98b1c57844cee2e0c WatchSource:0}: Error finding container 640ec633e9a874f27341ffef613b09a6535a4018175e78e98b1c57844cee2e0c: Status 404 returned error can't find the container with id 640ec633e9a874f27341ffef613b09a6535a4018175e78e98b1c57844cee2e0c Oct 30 21:40:24.560444 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:24.560422 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm78d\" (UniqueName: \"kubernetes.io/projected/0c851243-770c-4819-abb6-b19588f0fffe-kube-api-access-vm78d\") pod \"maas-api-revoke-sub-329ab292-q5xcq\" (UID: \"0c851243-770c-4819-abb6-b19588f0fffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-329ab292-q5xcq" Oct 30 21:40:24.673848 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:24.673760 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-329ab292-q5xcq" Oct 30 21:40:25.048212 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:25.048174 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-329ab292-q5xcq"] Oct 30 21:40:25.054088 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:40:25.054060 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c851243_770c_4819_abb6_b19588f0fffe.slice/crio-94047771e38eca69c9f5f34c11a0d7c3e314fa447c504ba51ec3b246766ee44a WatchSource:0}: Error finding container 94047771e38eca69c9f5f34c11a0d7c3e314fa447c504ba51ec3b246766ee44a: Status 404 returned error can't find the container with id 94047771e38eca69c9f5f34c11a0d7c3e314fa447c504ba51ec3b246766ee44a Oct 30 21:40:25.296265 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:25.296229 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c9ee410-qbfhf"] Oct 30 21:40:25.301200 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:25.301177 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9ee410-qbfhf" Oct 30 21:40:25.355223 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:25.355191 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c9ee410-qbfhf"] Oct 30 21:40:25.409107 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:25.409067 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz5qb\" (UniqueName: \"kubernetes.io/projected/b1f388b2-e1b0-4df6-b553-883a63f3401b-kube-api-access-cz5qb\") pod \"maas-api-revoke-sub-1c9ee410-qbfhf\" (UID: \"b1f388b2-e1b0-4df6-b553-883a63f3401b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9ee410-qbfhf" Oct 30 21:40:25.409300 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:25.409160 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f388b2-e1b0-4df6-b553-883a63f3401b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c9ee410-qbfhf\" (UID: \"b1f388b2-e1b0-4df6-b553-883a63f3401b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9ee410-qbfhf" Oct 30 21:40:25.409300 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:25.409268 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f388b2-e1b0-4df6-b553-883a63f3401b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c9ee410-qbfhf\" (UID: \"b1f388b2-e1b0-4df6-b553-883a63f3401b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9ee410-qbfhf" Oct 30 21:40:25.496287 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:25.496247 4560 generic.go:356] "Generic (PLEG): container finished" podID="49da1004-a76f-46f5-925d-a6a17be6d6e7" containerID="6146e0f05a9d978b43290b1d6cb83093286d8b71fd4a2bd1b33641d35ec3eee3" exitCode=0 Oct 30 21:40:25.496512 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:25.496360 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4327b99b-l5szt" event={"ID":"49da1004-a76f-46f5-925d-a6a17be6d6e7","Type":"ContainerDied","Data":"6146e0f05a9d978b43290b1d6cb83093286d8b71fd4a2bd1b33641d35ec3eee3"} Oct 30 21:40:25.496512 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:25.496416 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4327b99b-l5szt" event={"ID":"49da1004-a76f-46f5-925d-a6a17be6d6e7","Type":"ContainerStarted","Data":"640ec633e9a874f27341ffef613b09a6535a4018175e78e98b1c57844cee2e0c"} Oct 30 21:40:25.498018 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:25.497951 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-329ab292-q5xcq" event={"ID":"0c851243-770c-4819-abb6-b19588f0fffe","Type":"ContainerStarted","Data":"6832ec27c668e040bba774ffad4fc0d5dc0d67a9822e242ae356b4a109d994fe"} Oct 30 21:40:25.498018 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:25.497996 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-329ab292-q5xcq" event={"ID":"0c851243-770c-4819-abb6-b19588f0fffe","Type":"ContainerStarted","Data":"94047771e38eca69c9f5f34c11a0d7c3e314fa447c504ba51ec3b246766ee44a"} Oct 30 21:40:25.510572 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:25.510543 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f388b2-e1b0-4df6-b553-883a63f3401b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c9ee410-qbfhf\" (UID: \"b1f388b2-e1b0-4df6-b553-883a63f3401b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9ee410-qbfhf" Oct 30 21:40:25.510679 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:25.510628 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz5qb\" (UniqueName: \"kubernetes.io/projected/b1f388b2-e1b0-4df6-b553-883a63f3401b-kube-api-access-cz5qb\") pod \"maas-api-revoke-sub-1c9ee410-qbfhf\" (UID: \"b1f388b2-e1b0-4df6-b553-883a63f3401b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9ee410-qbfhf" Oct 30 21:40:25.510733 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:25.510677 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f388b2-e1b0-4df6-b553-883a63f3401b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c9ee410-qbfhf\" (UID: \"b1f388b2-e1b0-4df6-b553-883a63f3401b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9ee410-qbfhf" Oct 30 21:40:25.511292 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:25.511271 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f388b2-e1b0-4df6-b553-883a63f3401b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c9ee410-qbfhf\" (UID: \"b1f388b2-e1b0-4df6-b553-883a63f3401b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9ee410-qbfhf" Oct 30 21:40:25.513082 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:25.513062 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f388b2-e1b0-4df6-b553-883a63f3401b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c9ee410-qbfhf\" (UID: \"b1f388b2-e1b0-4df6-b553-883a63f3401b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9ee410-qbfhf" Oct 30 21:40:25.568779 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:25.568749 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz5qb\" (UniqueName: \"kubernetes.io/projected/b1f388b2-e1b0-4df6-b553-883a63f3401b-kube-api-access-cz5qb\") pod \"maas-api-revoke-sub-1c9ee410-qbfhf\" (UID: \"b1f388b2-e1b0-4df6-b553-883a63f3401b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9ee410-qbfhf" Oct 30 21:40:25.613476 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:25.613449 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9ee410-qbfhf" Oct 30 21:40:25.732376 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:25.732051 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-329ab292-q5xcq" podStartSLOduration=1.7320307160000001 podStartE2EDuration="1.732030716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:25.724886503 +0000 UTC m=+1601.279586281" watchObservedRunningTime="2026-10-30 21:40:25.732030716 +0000 UTC m=+1601.286730494" Oct 30 21:40:26.004897 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:26.004869 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c9ee410-qbfhf"] Oct 30 21:40:26.009925 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:40:26.009894 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1f388b2_e1b0_4df6_b553_883a63f3401b.slice/crio-d93a73ad711fae6fd1410a6c20abf037464c61fe99b76c500550bbedf2386741 WatchSource:0}: Error finding container d93a73ad711fae6fd1410a6c20abf037464c61fe99b76c500550bbedf2386741: Status 404 returned error can't find the container with id d93a73ad711fae6fd1410a6c20abf037464c61fe99b76c500550bbedf2386741 Oct 30 21:40:26.504979 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:26.504932 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9ee410-qbfhf" event={"ID":"b1f388b2-e1b0-4df6-b553-883a63f3401b","Type":"ContainerStarted","Data":"e53f80448d86d45eeb68d7aded90a60fdb1edc535a58bb6fd089ec81abf4568b"} Oct 30 21:40:26.505165 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:26.505001 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9ee410-qbfhf" event={"ID":"b1f388b2-e1b0-4df6-b553-883a63f3401b","Type":"ContainerStarted","Data":"d93a73ad711fae6fd1410a6c20abf037464c61fe99b76c500550bbedf2386741"} Oct 30 21:40:26.506531 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:26.506502 4560 generic.go:356] "Generic (PLEG): container finished" podID="0c851243-770c-4819-abb6-b19588f0fffe" containerID="6832ec27c668e040bba774ffad4fc0d5dc0d67a9822e242ae356b4a109d994fe" exitCode=0 Oct 30 21:40:26.506635 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:26.506587 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-329ab292-q5xcq" event={"ID":"0c851243-770c-4819-abb6-b19588f0fffe","Type":"ContainerDied","Data":"6832ec27c668e040bba774ffad4fc0d5dc0d67a9822e242ae356b4a109d994fe"} Oct 30 21:40:26.617524 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:26.617435 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9ee410-qbfhf" podStartSLOduration=1.617326205 podStartE2EDuration="1.617326205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:26.616464467 +0000 UTC m=+1602.171164255" watchObservedRunningTime="2026-10-30 21:40:26.617326205 +0000 UTC m=+1602.172025999" Oct 30 21:40:26.669526 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:26.669498 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4327b99b-l5szt" Oct 30 21:40:26.738236 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:26.738203 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49da1004-a76f-46f5-925d-a6a17be6d6e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49da1004-a76f-46f5-925d-a6a17be6d6e7-maas-api-openshift-ca\") pod \"49da1004-a76f-46f5-925d-a6a17be6d6e7\" (UID: \"49da1004-a76f-46f5-925d-a6a17be6d6e7\") " Oct 30 21:40:26.738729 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:26.738298 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49da1004-a76f-46f5-925d-a6a17be6d6e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49da1004-a76f-46f5-925d-a6a17be6d6e7-maas-api-xks-ca\") pod \"49da1004-a76f-46f5-925d-a6a17be6d6e7\" (UID: \"49da1004-a76f-46f5-925d-a6a17be6d6e7\") " Oct 30 21:40:26.738729 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:26.738323 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49da1004-a76f-46f5-925d-a6a17be6d6e7-kube-api-access-b9nqf\" (UniqueName: \"kubernetes.io/projected/49da1004-a76f-46f5-925d-a6a17be6d6e7-kube-api-access-b9nqf\") pod \"49da1004-a76f-46f5-925d-a6a17be6d6e7\" (UID: \"49da1004-a76f-46f5-925d-a6a17be6d6e7\") " Oct 30 21:40:26.738729 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:26.738579 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49da1004-a76f-46f5-925d-a6a17be6d6e7-maas-api-openshift-ca" pod "49da1004-a76f-46f5-925d-a6a17be6d6e7" (UID: "49da1004-a76f-46f5-925d-a6a17be6d6e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:26.740433 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:26.740409 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49da1004-a76f-46f5-925d-a6a17be6d6e7-kube-api-access-b9nqf" pod "49da1004-a76f-46f5-925d-a6a17be6d6e7" (UID: "49da1004-a76f-46f5-925d-a6a17be6d6e7"). InnerVolumeSpecName "kube-api-access-b9nqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:26.740433 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:26.740417 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49da1004-a76f-46f5-925d-a6a17be6d6e7-maas-api-xks-ca" pod "49da1004-a76f-46f5-925d-a6a17be6d6e7" (UID: "49da1004-a76f-46f5-925d-a6a17be6d6e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:26.839698 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:26.839655 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49da1004-a76f-46f5-925d-a6a17be6d6e7-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:26.839698 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:26.839694 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49da1004-a76f-46f5-925d-a6a17be6d6e7-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:26.839698 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:26.839707 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9nqf\" (UniqueName: \"kubernetes.io/projected/49da1004-a76f-46f5-925d-a6a17be6d6e7-kube-api-access-b9nqf\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:27.519295 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:27.519266 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4327b99b-l5szt" Oct 30 21:40:27.519519 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:27.519299 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4327b99b-l5szt" event={"ID":"49da1004-a76f-46f5-925d-a6a17be6d6e7","Type":"ContainerDied","Data":"640ec633e9a874f27341ffef613b09a6535a4018175e78e98b1c57844cee2e0c"} Oct 30 21:40:27.519519 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:27.519405 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="640ec633e9a874f27341ffef613b09a6535a4018175e78e98b1c57844cee2e0c" Oct 30 21:40:27.520851 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:27.520822 4560 generic.go:356] "Generic (PLEG): container finished" podID="b1f388b2-e1b0-4df6-b553-883a63f3401b" containerID="e53f80448d86d45eeb68d7aded90a60fdb1edc535a58bb6fd089ec81abf4568b" exitCode=0 Oct 30 21:40:27.520956 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:27.520890 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9ee410-qbfhf" event={"ID":"b1f388b2-e1b0-4df6-b553-883a63f3401b","Type":"ContainerDied","Data":"e53f80448d86d45eeb68d7aded90a60fdb1edc535a58bb6fd089ec81abf4568b"} Oct 30 21:40:27.658945 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:27.658922 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-329ab292-q5xcq" Oct 30 21:40:27.749919 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:27.749882 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c851243-770c-4819-abb6-b19588f0fffe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c851243-770c-4819-abb6-b19588f0fffe-maas-api-openshift-ca\") pod \"0c851243-770c-4819-abb6-b19588f0fffe\" (UID: \"0c851243-770c-4819-abb6-b19588f0fffe\") " Oct 30 21:40:27.750395 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:27.749938 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c851243-770c-4819-abb6-b19588f0fffe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c851243-770c-4819-abb6-b19588f0fffe-maas-api-xks-ca\") pod \"0c851243-770c-4819-abb6-b19588f0fffe\" (UID: \"0c851243-770c-4819-abb6-b19588f0fffe\") " Oct 30 21:40:27.750395 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:27.749974 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c851243-770c-4819-abb6-b19588f0fffe-kube-api-access-vm78d\" (UniqueName: \"kubernetes.io/projected/0c851243-770c-4819-abb6-b19588f0fffe-kube-api-access-vm78d\") pod \"0c851243-770c-4819-abb6-b19588f0fffe\" (UID: \"0c851243-770c-4819-abb6-b19588f0fffe\") " Oct 30 21:40:27.750395 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:27.750251 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c851243-770c-4819-abb6-b19588f0fffe-maas-api-openshift-ca" pod "0c851243-770c-4819-abb6-b19588f0fffe" (UID: "0c851243-770c-4819-abb6-b19588f0fffe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:27.750518 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:27.750425 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c851243-770c-4819-abb6-b19588f0fffe-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:27.752000 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:27.751976 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c851243-770c-4819-abb6-b19588f0fffe-kube-api-access-vm78d" pod "0c851243-770c-4819-abb6-b19588f0fffe" (UID: "0c851243-770c-4819-abb6-b19588f0fffe"). InnerVolumeSpecName "kube-api-access-vm78d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:27.752109 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:27.752090 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c851243-770c-4819-abb6-b19588f0fffe-maas-api-xks-ca" pod "0c851243-770c-4819-abb6-b19588f0fffe" (UID: "0c851243-770c-4819-abb6-b19588f0fffe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:27.851599 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:27.851528 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c851243-770c-4819-abb6-b19588f0fffe-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:27.851599 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:27.851554 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm78d\" (UniqueName: \"kubernetes.io/projected/0c851243-770c-4819-abb6-b19588f0fffe-kube-api-access-vm78d\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:28.526702 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:28.526673 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-329ab292-q5xcq" Oct 30 21:40:28.526900 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:28.526705 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-329ab292-q5xcq" event={"ID":"0c851243-770c-4819-abb6-b19588f0fffe","Type":"ContainerDied","Data":"94047771e38eca69c9f5f34c11a0d7c3e314fa447c504ba51ec3b246766ee44a"} Oct 30 21:40:28.526900 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:28.526751 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94047771e38eca69c9f5f34c11a0d7c3e314fa447c504ba51ec3b246766ee44a" Oct 30 21:40:28.663448 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:28.663424 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9ee410-qbfhf" Oct 30 21:40:28.760701 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:28.760666 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1f388b2-e1b0-4df6-b553-883a63f3401b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f388b2-e1b0-4df6-b553-883a63f3401b-maas-api-openshift-ca\") pod \"b1f388b2-e1b0-4df6-b553-883a63f3401b\" (UID: \"b1f388b2-e1b0-4df6-b553-883a63f3401b\") " Oct 30 21:40:28.761122 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:28.760729 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1f388b2-e1b0-4df6-b553-883a63f3401b-kube-api-access-cz5qb\" (UniqueName: \"kubernetes.io/projected/b1f388b2-e1b0-4df6-b553-883a63f3401b-kube-api-access-cz5qb\") pod \"b1f388b2-e1b0-4df6-b553-883a63f3401b\" (UID: \"b1f388b2-e1b0-4df6-b553-883a63f3401b\") " Oct 30 21:40:28.761122 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:28.760852 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1f388b2-e1b0-4df6-b553-883a63f3401b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f388b2-e1b0-4df6-b553-883a63f3401b-maas-api-xks-ca\") pod \"b1f388b2-e1b0-4df6-b553-883a63f3401b\" (UID: \"b1f388b2-e1b0-4df6-b553-883a63f3401b\") " Oct 30 21:40:28.761122 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:28.761058 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1f388b2-e1b0-4df6-b553-883a63f3401b-maas-api-openshift-ca" pod "b1f388b2-e1b0-4df6-b553-883a63f3401b" (UID: "b1f388b2-e1b0-4df6-b553-883a63f3401b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:28.761300 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:28.761175 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f388b2-e1b0-4df6-b553-883a63f3401b-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:28.762910 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:28.762883 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1f388b2-e1b0-4df6-b553-883a63f3401b-maas-api-xks-ca" pod "b1f388b2-e1b0-4df6-b553-883a63f3401b" (UID: "b1f388b2-e1b0-4df6-b553-883a63f3401b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:28.762910 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:28.762901 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1f388b2-e1b0-4df6-b553-883a63f3401b-kube-api-access-cz5qb" pod "b1f388b2-e1b0-4df6-b553-883a63f3401b" (UID: "b1f388b2-e1b0-4df6-b553-883a63f3401b"). InnerVolumeSpecName "kube-api-access-cz5qb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:28.862148 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:28.862073 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f388b2-e1b0-4df6-b553-883a63f3401b-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:28.862148 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:28.862100 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz5qb\" (UniqueName: \"kubernetes.io/projected/b1f388b2-e1b0-4df6-b553-883a63f3401b-kube-api-access-cz5qb\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:29.464256 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.464218 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd"] Oct 30 21:40:29.464604 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.464590 4560 cpu_manager.go:414] "containerMap: removing container" podUID="0c851243-770c-4819-abb6-b19588f0fffe" containerName="revoke-keys" Oct 30 21:40:29.464658 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.464605 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c851243-770c-4819-abb6-b19588f0fffe" containerName="revoke-keys" Oct 30 21:40:29.464658 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.464614 4560 cpu_manager.go:414] "containerMap: removing container" podUID="b1f388b2-e1b0-4df6-b553-883a63f3401b" containerName="revoke-keys" Oct 30 21:40:29.464658 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.464619 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1f388b2-e1b0-4df6-b553-883a63f3401b" containerName="revoke-keys" Oct 30 21:40:29.464658 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.464653 4560 cpu_manager.go:414] "containerMap: removing container" podUID="49da1004-a76f-46f5-925d-a6a17be6d6e7" containerName="revoke-keys" Oct 30 21:40:29.464775 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.464661 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49da1004-a76f-46f5-925d-a6a17be6d6e7" containerName="revoke-keys" Oct 30 21:40:29.464775 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.464724 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="49da1004-a76f-46f5-925d-a6a17be6d6e7" containerName="revoke-keys" Oct 30 21:40:29.464775 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.464734 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c851243-770c-4819-abb6-b19588f0fffe" containerName="revoke-keys" Oct 30 21:40:29.464775 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.464739 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1f388b2-e1b0-4df6-b553-883a63f3401b" containerName="revoke-keys" Oct 30 21:40:29.467974 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.467949 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" Oct 30 21:40:29.519429 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.519404 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-afbe5b\"/\"distinct-w0-afbe5b-kserve-self-signed-certs\"" Oct 30 21:40:29.535076 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.535048 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9ee410-qbfhf" event={"ID":"b1f388b2-e1b0-4df6-b553-883a63f3401b","Type":"ContainerDied","Data":"d93a73ad711fae6fd1410a6c20abf037464c61fe99b76c500550bbedf2386741"} Oct 30 21:40:29.535207 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.535077 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9ee410-qbfhf" Oct 30 21:40:29.535207 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.535093 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d93a73ad711fae6fd1410a6c20abf037464c61fe99b76c500550bbedf2386741" Oct 30 21:40:29.568511 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.568480 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd"] Oct 30 21:40:29.568838 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.568818 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-dshm\") pod \"distinct-w0-afbe5b-kserve-9f845d77b-kftqd\" (UID: \"971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d\") " pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" Oct 30 21:40:29.568901 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.568864 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-home\") pod \"distinct-w0-afbe5b-kserve-9f845d77b-kftqd\" (UID: \"971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d\") " pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" Oct 30 21:40:29.568901 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.568879 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-model-cache\") pod \"distinct-w0-afbe5b-kserve-9f845d77b-kftqd\" (UID: \"971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d\") " pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" Oct 30 21:40:29.569013 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.568995 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-tmp-dir\") pod \"distinct-w0-afbe5b-kserve-9f845d77b-kftqd\" (UID: \"971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d\") " pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" Oct 30 21:40:29.569048 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.569042 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-tls-certs\") pod \"distinct-w0-afbe5b-kserve-9f845d77b-kftqd\" (UID: \"971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d\") " pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" Oct 30 21:40:29.569241 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.569223 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5452f\" (UniqueName: \"kubernetes.io/projected/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-kube-api-access-5452f\") pod \"distinct-w0-afbe5b-kserve-9f845d77b-kftqd\" (UID: \"971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d\") " pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" Oct 30 21:40:29.670211 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.670170 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-tls-certs\") pod \"distinct-w0-afbe5b-kserve-9f845d77b-kftqd\" (UID: \"971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d\") " pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" Oct 30 21:40:29.670410 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.670270 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5452f\" (UniqueName: \"kubernetes.io/projected/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-kube-api-access-5452f\") pod \"distinct-w0-afbe5b-kserve-9f845d77b-kftqd\" (UID: \"971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d\") " pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" Oct 30 21:40:29.670410 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.670316 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-dshm\") pod \"distinct-w0-afbe5b-kserve-9f845d77b-kftqd\" (UID: \"971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d\") " pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" Oct 30 21:40:29.670410 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.670381 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-home\") pod \"distinct-w0-afbe5b-kserve-9f845d77b-kftqd\" (UID: \"971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d\") " pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" Oct 30 21:40:29.670410 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.670404 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-model-cache\") pod \"distinct-w0-afbe5b-kserve-9f845d77b-kftqd\" (UID: \"971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d\") " pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" Oct 30 21:40:29.670625 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.670432 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-tmp-dir\") pod \"distinct-w0-afbe5b-kserve-9f845d77b-kftqd\" (UID: \"971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d\") " pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" Oct 30 21:40:29.670822 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.670802 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-tmp-dir\") pod \"distinct-w0-afbe5b-kserve-9f845d77b-kftqd\" (UID: \"971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d\") " pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" Oct 30 21:40:29.670911 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.670888 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-home\") pod \"distinct-w0-afbe5b-kserve-9f845d77b-kftqd\" (UID: \"971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d\") " pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" Oct 30 21:40:29.671009 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.670992 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-model-cache\") pod \"distinct-w0-afbe5b-kserve-9f845d77b-kftqd\" (UID: \"971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d\") " pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" Oct 30 21:40:29.672617 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.672601 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-dshm\") pod \"distinct-w0-afbe5b-kserve-9f845d77b-kftqd\" (UID: \"971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d\") " pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" Oct 30 21:40:29.673136 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.673109 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-tls-certs\") pod \"distinct-w0-afbe5b-kserve-9f845d77b-kftqd\" (UID: \"971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d\") " pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" Oct 30 21:40:29.728133 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.728107 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5452f\" (UniqueName: \"kubernetes.io/projected/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-kube-api-access-5452f\") pod \"distinct-w0-afbe5b-kserve-9f845d77b-kftqd\" (UID: \"971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d\") " pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" Oct 30 21:40:29.780149 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.780113 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" Oct 30 21:40:29.839433 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.838971 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw"] Oct 30 21:40:29.874692 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.874662 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" Oct 30 21:40:29.931266 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.931227 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw"] Oct 30 21:40:29.975673 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.975638 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34448c97-8c89-4c69-9e3b-8f513d88b803-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-pcphw\" (UID: \"34448c97-8c89-4c69-9e3b-8f513d88b803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" Oct 30 21:40:29.975853 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.975687 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34448c97-8c89-4c69-9e3b-8f513d88b803-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pcphw\" (UID: \"34448c97-8c89-4c69-9e3b-8f513d88b803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" Oct 30 21:40:29.975853 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.975775 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34448c97-8c89-4c69-9e3b-8f513d88b803-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-pcphw\" (UID: \"34448c97-8c89-4c69-9e3b-8f513d88b803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" Oct 30 21:40:29.975853 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.975821 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34448c97-8c89-4c69-9e3b-8f513d88b803-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-pcphw\" (UID: \"34448c97-8c89-4c69-9e3b-8f513d88b803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" Oct 30 21:40:29.975995 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.975885 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34448c97-8c89-4c69-9e3b-8f513d88b803-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-pcphw\" (UID: \"34448c97-8c89-4c69-9e3b-8f513d88b803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" Oct 30 21:40:29.975995 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.975925 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34448c97-8c89-4c69-9e3b-8f513d88b803-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-pcphw\" (UID: \"34448c97-8c89-4c69-9e3b-8f513d88b803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" Oct 30 21:40:29.976067 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.976000 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqk7t\" (UniqueName: \"kubernetes.io/projected/34448c97-8c89-4c69-9e3b-8f513d88b803-kube-api-access-fqk7t\") pod \"maas-default-gateway-openshift-default-67bf986f99-pcphw\" (UID: \"34448c97-8c89-4c69-9e3b-8f513d88b803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" Oct 30 21:40:29.976067 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.976032 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34448c97-8c89-4c69-9e3b-8f513d88b803-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pcphw\" (UID: \"34448c97-8c89-4c69-9e3b-8f513d88b803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" Oct 30 21:40:29.976067 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:29.976058 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34448c97-8c89-4c69-9e3b-8f513d88b803-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-pcphw\" (UID: \"34448c97-8c89-4c69-9e3b-8f513d88b803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" Oct 30 21:40:30.076810 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:30.076737 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqk7t\" (UniqueName: \"kubernetes.io/projected/34448c97-8c89-4c69-9e3b-8f513d88b803-kube-api-access-fqk7t\") pod \"maas-default-gateway-openshift-default-67bf986f99-pcphw\" (UID: \"34448c97-8c89-4c69-9e3b-8f513d88b803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" Oct 30 21:40:30.076810 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:30.076779 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34448c97-8c89-4c69-9e3b-8f513d88b803-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pcphw\" (UID: \"34448c97-8c89-4c69-9e3b-8f513d88b803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" Oct 30 21:40:30.076810 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:30.076805 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34448c97-8c89-4c69-9e3b-8f513d88b803-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-pcphw\" (UID: \"34448c97-8c89-4c69-9e3b-8f513d88b803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" Oct 30 21:40:30.077022 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:30.076841 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34448c97-8c89-4c69-9e3b-8f513d88b803-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-pcphw\" (UID: \"34448c97-8c89-4c69-9e3b-8f513d88b803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" Oct 30 21:40:30.077022 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:30.076872 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34448c97-8c89-4c69-9e3b-8f513d88b803-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pcphw\" (UID: \"34448c97-8c89-4c69-9e3b-8f513d88b803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" Oct 30 21:40:30.077022 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:30.076895 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34448c97-8c89-4c69-9e3b-8f513d88b803-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-pcphw\" (UID: \"34448c97-8c89-4c69-9e3b-8f513d88b803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" Oct 30 21:40:30.077168 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:30.077090 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34448c97-8c89-4c69-9e3b-8f513d88b803-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-pcphw\" (UID: \"34448c97-8c89-4c69-9e3b-8f513d88b803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" Oct 30 21:40:30.077168 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:30.077146 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34448c97-8c89-4c69-9e3b-8f513d88b803-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-pcphw\" (UID: \"34448c97-8c89-4c69-9e3b-8f513d88b803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" Oct 30 21:40:30.077262 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:30.077176 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34448c97-8c89-4c69-9e3b-8f513d88b803-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pcphw\" (UID: \"34448c97-8c89-4c69-9e3b-8f513d88b803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" Oct 30 21:40:30.077262 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:30.077235 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34448c97-8c89-4c69-9e3b-8f513d88b803-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pcphw\" (UID: \"34448c97-8c89-4c69-9e3b-8f513d88b803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" Oct 30 21:40:30.077390 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:30.077273 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34448c97-8c89-4c69-9e3b-8f513d88b803-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-pcphw\" (UID: \"34448c97-8c89-4c69-9e3b-8f513d88b803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" Oct 30 21:40:30.077461 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:30.077430 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34448c97-8c89-4c69-9e3b-8f513d88b803-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-pcphw\" (UID: \"34448c97-8c89-4c69-9e3b-8f513d88b803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" Oct 30 21:40:30.077613 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:30.077594 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34448c97-8c89-4c69-9e3b-8f513d88b803-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-pcphw\" (UID: \"34448c97-8c89-4c69-9e3b-8f513d88b803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" Oct 30 21:40:30.077662 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:30.077599 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34448c97-8c89-4c69-9e3b-8f513d88b803-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-pcphw\" (UID: \"34448c97-8c89-4c69-9e3b-8f513d88b803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" Oct 30 21:40:30.079110 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:30.079086 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34448c97-8c89-4c69-9e3b-8f513d88b803-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-pcphw\" (UID: \"34448c97-8c89-4c69-9e3b-8f513d88b803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" Oct 30 21:40:30.079397 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:30.079381 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34448c97-8c89-4c69-9e3b-8f513d88b803-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-pcphw\" (UID: \"34448c97-8c89-4c69-9e3b-8f513d88b803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" Oct 30 21:40:30.135750 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:30.135725 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34448c97-8c89-4c69-9e3b-8f513d88b803-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-pcphw\" (UID: \"34448c97-8c89-4c69-9e3b-8f513d88b803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" Oct 30 21:40:30.135856 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:30.135827 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqk7t\" (UniqueName: \"kubernetes.io/projected/34448c97-8c89-4c69-9e3b-8f513d88b803-kube-api-access-fqk7t\") pod \"maas-default-gateway-openshift-default-67bf986f99-pcphw\" (UID: \"34448c97-8c89-4c69-9e3b-8f513d88b803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" Oct 30 21:40:30.157419 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:30.157392 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd"] Oct 30 21:40:30.162606 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:40:30.162583 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod971ed2b8_e2df_4df0_a240_4c3f0c0d7c0d.slice/crio-2fc84004c095b341be3c9751697b2a36f3988d4a44abd09981bcb1e7157ed978 WatchSource:0}: Error finding container 2fc84004c095b341be3c9751697b2a36f3988d4a44abd09981bcb1e7157ed978: Status 404 returned error can't find the container with id 2fc84004c095b341be3c9751697b2a36f3988d4a44abd09981bcb1e7157ed978 Oct 30 21:40:30.208489 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:30.208459 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" Oct 30 21:40:30.541931 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:30.541895 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" event={"ID":"971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d","Type":"ContainerStarted","Data":"3b09ed402efdf45ad09ae6b962dcff9a0c1711d4707e88e95372afeac16abebc"} Oct 30 21:40:30.542088 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:30.541940 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" event={"ID":"971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d","Type":"ContainerStarted","Data":"2fc84004c095b341be3c9751697b2a36f3988d4a44abd09981bcb1e7157ed978"} Oct 30 21:40:30.642727 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:30.642695 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw"] Oct 30 21:40:30.648953 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:40:30.648927 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34448c97_8c89_4c69_9e3b_8f513d88b803.slice/crio-9d77c7042cb49ec1e057d977c84e79a1258c7a602a183136226175d01ac3cf2f WatchSource:0}: Error finding container 9d77c7042cb49ec1e057d977c84e79a1258c7a602a183136226175d01ac3cf2f: Status 404 returned error can't find the container with id 9d77c7042cb49ec1e057d977c84e79a1258c7a602a183136226175d01ac3cf2f Oct 30 21:40:30.651129 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:30.651100 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:40:30.651194 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:30.651156 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:40:30.651194 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:30.651186 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:40:30.687952 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:30.687906 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" podStartSLOduration=1.687893829 podStartE2EDuration="1.687893829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:30.686873371 +0000 UTC m=+1606.241573149" watchObservedRunningTime="2026-10-30 21:40:30.687893829 +0000 UTC m=+1606.242593606" Oct 30 21:40:30.782859 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:30.782814 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" Oct 30 21:40:31.554062 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:31.553752 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" event={"ID":"34448c97-8c89-4c69-9e3b-8f513d88b803","Type":"ContainerStarted","Data":"8944a3b4f6fbac741bd92c01c7ac681a52de24ddde7774811b24688f15708db0"} Oct 30 21:40:31.554062 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:31.553818 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" event={"ID":"34448c97-8c89-4c69-9e3b-8f513d88b803","Type":"ContainerStarted","Data":"9d77c7042cb49ec1e057d977c84e79a1258c7a602a183136226175d01ac3cf2f"} Oct 30 21:40:31.716190 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:31.716138 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" podStartSLOduration=2.71612398 podStartE2EDuration="2.71612398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:31.715004493 +0000 UTC m=+1607.269704271" watchObservedRunningTime="2026-10-30 21:40:31.71612398 +0000 UTC m=+1607.270823758" Oct 30 21:40:32.209181 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:32.209148 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" Oct 30 21:40:32.210711 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:32.210688 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pcphw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:40:32.210815 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:32.210734 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" podUID="34448c97-8c89-4c69-9e3b-8f513d88b803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 21:40:33.209421 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:33.209384 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pcphw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:40:33.209833 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:33.209445 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" podUID="34448c97-8c89-4c69-9e3b-8f513d88b803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 21:40:33.696164 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:33.696134 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84fa9d95-csxcd"] Oct 30 21:40:33.700752 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:33.700733 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fa9d95-csxcd" Oct 30 21:40:33.751695 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:33.751652 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:40:33.798406 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:33.798376 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:40:33.798406 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:33.798390 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:40:33.798406 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:33.798390 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:40:33.803198 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:33.803168 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84fa9d95-csxcd"] Oct 30 21:40:33.817917 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:33.817894 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba20dae7-d119-48cb-9f68-f8619ebd303b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84fa9d95-csxcd\" (UID: \"ba20dae7-d119-48cb-9f68-f8619ebd303b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fa9d95-csxcd" Oct 30 21:40:33.818038 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:33.817931 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgld9\" (UniqueName: \"kubernetes.io/projected/ba20dae7-d119-48cb-9f68-f8619ebd303b-kube-api-access-qgld9\") pod \"maas-api-revoke-sub-84fa9d95-csxcd\" (UID: \"ba20dae7-d119-48cb-9f68-f8619ebd303b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fa9d95-csxcd" Oct 30 21:40:33.818038 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:33.817965 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba20dae7-d119-48cb-9f68-f8619ebd303b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84fa9d95-csxcd\" (UID: \"ba20dae7-d119-48cb-9f68-f8619ebd303b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fa9d95-csxcd" Oct 30 21:40:33.919285 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:33.919257 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba20dae7-d119-48cb-9f68-f8619ebd303b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84fa9d95-csxcd\" (UID: \"ba20dae7-d119-48cb-9f68-f8619ebd303b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fa9d95-csxcd" Oct 30 21:40:33.919461 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:33.919296 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgld9\" (UniqueName: \"kubernetes.io/projected/ba20dae7-d119-48cb-9f68-f8619ebd303b-kube-api-access-qgld9\") pod \"maas-api-revoke-sub-84fa9d95-csxcd\" (UID: \"ba20dae7-d119-48cb-9f68-f8619ebd303b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fa9d95-csxcd" Oct 30 21:40:33.919461 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:33.919333 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba20dae7-d119-48cb-9f68-f8619ebd303b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84fa9d95-csxcd\" (UID: \"ba20dae7-d119-48cb-9f68-f8619ebd303b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fa9d95-csxcd" Oct 30 21:40:33.920003 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:33.919975 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba20dae7-d119-48cb-9f68-f8619ebd303b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84fa9d95-csxcd\" (UID: \"ba20dae7-d119-48cb-9f68-f8619ebd303b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fa9d95-csxcd" Oct 30 21:40:33.921582 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:33.921562 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba20dae7-d119-48cb-9f68-f8619ebd303b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84fa9d95-csxcd\" (UID: \"ba20dae7-d119-48cb-9f68-f8619ebd303b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fa9d95-csxcd" Oct 30 21:40:33.978091 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:33.978053 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgld9\" (UniqueName: \"kubernetes.io/projected/ba20dae7-d119-48cb-9f68-f8619ebd303b-kube-api-access-qgld9\") pod \"maas-api-revoke-sub-84fa9d95-csxcd\" (UID: \"ba20dae7-d119-48cb-9f68-f8619ebd303b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fa9d95-csxcd" Oct 30 21:40:34.013327 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:34.013298 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fa9d95-csxcd" Oct 30 21:40:34.209222 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:34.209186 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pcphw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:40:34.209407 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:34.209238 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" podUID="34448c97-8c89-4c69-9e3b-8f513d88b803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 21:40:34.406477 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:34.406398 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84fa9d95-csxcd"] Oct 30 21:40:34.412605 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:40:34.412578 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba20dae7_d119_48cb_9f68_f8619ebd303b.slice/crio-012730ea1b758da898ab4103e60efe03fef9ca3d79511eebfe2a50dbecb8ba81 WatchSource:0}: Error finding container 012730ea1b758da898ab4103e60efe03fef9ca3d79511eebfe2a50dbecb8ba81: Status 404 returned error can't find the container with id 012730ea1b758da898ab4103e60efe03fef9ca3d79511eebfe2a50dbecb8ba81 Oct 30 21:40:34.575268 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:34.575231 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fa9d95-csxcd" event={"ID":"ba20dae7-d119-48cb-9f68-f8619ebd303b","Type":"ContainerStarted","Data":"012730ea1b758da898ab4103e60efe03fef9ca3d79511eebfe2a50dbecb8ba81"} Oct 30 21:40:34.756578 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:34.756541 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5d8ac1a-6jgw5"] Oct 30 21:40:34.760796 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:34.760780 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d8ac1a-6jgw5" Oct 30 21:40:34.814164 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:34.814134 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5d8ac1a-6jgw5"] Oct 30 21:40:34.844361 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:34.838890 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a42988c9-a7e6-4dde-88d9-73ea4586dd79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5d8ac1a-6jgw5\" (UID: \"a42988c9-a7e6-4dde-88d9-73ea4586dd79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d8ac1a-6jgw5" Oct 30 21:40:34.844361 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:34.838946 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a42988c9-a7e6-4dde-88d9-73ea4586dd79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5d8ac1a-6jgw5\" (UID: \"a42988c9-a7e6-4dde-88d9-73ea4586dd79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d8ac1a-6jgw5" Oct 30 21:40:34.844361 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:34.838993 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjtrq\" (UniqueName: \"kubernetes.io/projected/a42988c9-a7e6-4dde-88d9-73ea4586dd79-kube-api-access-cjtrq\") pod \"maas-api-revoke-sub-b5d8ac1a-6jgw5\" (UID: \"a42988c9-a7e6-4dde-88d9-73ea4586dd79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d8ac1a-6jgw5" Oct 30 21:40:34.948784 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:34.947943 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a42988c9-a7e6-4dde-88d9-73ea4586dd79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5d8ac1a-6jgw5\" (UID: \"a42988c9-a7e6-4dde-88d9-73ea4586dd79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d8ac1a-6jgw5" Oct 30 21:40:34.948784 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:34.948000 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a42988c9-a7e6-4dde-88d9-73ea4586dd79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5d8ac1a-6jgw5\" (UID: \"a42988c9-a7e6-4dde-88d9-73ea4586dd79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d8ac1a-6jgw5" Oct 30 21:40:34.948784 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:34.948054 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjtrq\" (UniqueName: \"kubernetes.io/projected/a42988c9-a7e6-4dde-88d9-73ea4586dd79-kube-api-access-cjtrq\") pod \"maas-api-revoke-sub-b5d8ac1a-6jgw5\" (UID: \"a42988c9-a7e6-4dde-88d9-73ea4586dd79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d8ac1a-6jgw5" Oct 30 21:40:34.949258 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:34.949124 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a42988c9-a7e6-4dde-88d9-73ea4586dd79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5d8ac1a-6jgw5\" (UID: \"a42988c9-a7e6-4dde-88d9-73ea4586dd79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d8ac1a-6jgw5" Oct 30 21:40:34.953753 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:34.953732 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a42988c9-a7e6-4dde-88d9-73ea4586dd79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5d8ac1a-6jgw5\" (UID: \"a42988c9-a7e6-4dde-88d9-73ea4586dd79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d8ac1a-6jgw5" Oct 30 21:40:35.005968 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:35.005942 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjtrq\" (UniqueName: \"kubernetes.io/projected/a42988c9-a7e6-4dde-88d9-73ea4586dd79-kube-api-access-cjtrq\") pod \"maas-api-revoke-sub-b5d8ac1a-6jgw5\" (UID: \"a42988c9-a7e6-4dde-88d9-73ea4586dd79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d8ac1a-6jgw5" Oct 30 21:40:35.073360 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:35.073316 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d8ac1a-6jgw5" Oct 30 21:40:35.209038 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:35.208964 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pcphw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:40:35.209038 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:35.209017 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" podUID="34448c97-8c89-4c69-9e3b-8f513d88b803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 21:40:35.460717 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:35.460650 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5d8ac1a-6jgw5"] Oct 30 21:40:35.478243 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:40:35.478211 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda42988c9_a7e6_4dde_88d9_73ea4586dd79.slice/crio-db3a277e002444a19405fdda0d3820791d60161841669e4872ed666eebf950c3 WatchSource:0}: Error finding container db3a277e002444a19405fdda0d3820791d60161841669e4872ed666eebf950c3: Status 404 returned error can't find the container with id db3a277e002444a19405fdda0d3820791d60161841669e4872ed666eebf950c3 Oct 30 21:40:35.578631 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:35.578597 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d8ac1a-6jgw5" event={"ID":"a42988c9-a7e6-4dde-88d9-73ea4586dd79","Type":"ContainerStarted","Data":"db3a277e002444a19405fdda0d3820791d60161841669e4872ed666eebf950c3"} Oct 30 21:40:35.580124 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:35.580093 4560 generic.go:356] "Generic (PLEG): container finished" podID="ba20dae7-d119-48cb-9f68-f8619ebd303b" containerID="3ad254a96d9b853c2d3c74699e26b21d96712a20b0281bc572f4fba5e83d0515" exitCode=0 Oct 30 21:40:35.580239 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:35.580130 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fa9d95-csxcd" event={"ID":"ba20dae7-d119-48cb-9f68-f8619ebd303b","Type":"ContainerDied","Data":"3ad254a96d9b853c2d3c74699e26b21d96712a20b0281bc572f4fba5e83d0515"} Oct 30 21:40:36.209253 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:36.209172 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pcphw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:40:36.209253 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:36.209224 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" podUID="34448c97-8c89-4c69-9e3b-8f513d88b803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 21:40:36.587926 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:36.587884 4560 generic.go:356] "Generic (PLEG): container finished" podID="a42988c9-a7e6-4dde-88d9-73ea4586dd79" containerID="c257fc7912bb943b0e68d5a49d31eac41d59c429e2f2560c2b51ded88dd7adfd" exitCode=0 Oct 30 21:40:36.588332 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:36.587944 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d8ac1a-6jgw5" event={"ID":"a42988c9-a7e6-4dde-88d9-73ea4586dd79","Type":"ContainerDied","Data":"c257fc7912bb943b0e68d5a49d31eac41d59c429e2f2560c2b51ded88dd7adfd"} Oct 30 21:40:36.731650 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:36.731628 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fa9d95-csxcd" Oct 30 21:40:36.875147 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:36.875065 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba20dae7-d119-48cb-9f68-f8619ebd303b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba20dae7-d119-48cb-9f68-f8619ebd303b-maas-api-openshift-ca\") pod \"ba20dae7-d119-48cb-9f68-f8619ebd303b\" (UID: \"ba20dae7-d119-48cb-9f68-f8619ebd303b\") " Oct 30 21:40:36.875313 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:36.875150 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba20dae7-d119-48cb-9f68-f8619ebd303b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba20dae7-d119-48cb-9f68-f8619ebd303b-maas-api-xks-ca\") pod \"ba20dae7-d119-48cb-9f68-f8619ebd303b\" (UID: \"ba20dae7-d119-48cb-9f68-f8619ebd303b\") " Oct 30 21:40:36.875313 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:36.875172 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba20dae7-d119-48cb-9f68-f8619ebd303b-kube-api-access-qgld9\" (UniqueName: \"kubernetes.io/projected/ba20dae7-d119-48cb-9f68-f8619ebd303b-kube-api-access-qgld9\") pod \"ba20dae7-d119-48cb-9f68-f8619ebd303b\" (UID: \"ba20dae7-d119-48cb-9f68-f8619ebd303b\") " Oct 30 21:40:36.875605 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:36.875567 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba20dae7-d119-48cb-9f68-f8619ebd303b-maas-api-openshift-ca" pod "ba20dae7-d119-48cb-9f68-f8619ebd303b" (UID: "ba20dae7-d119-48cb-9f68-f8619ebd303b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:36.877372 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:36.877319 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba20dae7-d119-48cb-9f68-f8619ebd303b-maas-api-xks-ca" pod "ba20dae7-d119-48cb-9f68-f8619ebd303b" (UID: "ba20dae7-d119-48cb-9f68-f8619ebd303b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:36.877455 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:36.877371 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba20dae7-d119-48cb-9f68-f8619ebd303b-kube-api-access-qgld9" pod "ba20dae7-d119-48cb-9f68-f8619ebd303b" (UID: "ba20dae7-d119-48cb-9f68-f8619ebd303b"). InnerVolumeSpecName "kube-api-access-qgld9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:36.976271 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:36.976240 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba20dae7-d119-48cb-9f68-f8619ebd303b-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:36.976271 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:36.976269 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba20dae7-d119-48cb-9f68-f8619ebd303b-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:36.976454 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:36.976283 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgld9\" (UniqueName: \"kubernetes.io/projected/ba20dae7-d119-48cb-9f68-f8619ebd303b-kube-api-access-qgld9\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:37.209269 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:37.209190 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pcphw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:40:37.209269 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:37.209247 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" podUID="34448c97-8c89-4c69-9e3b-8f513d88b803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 21:40:37.593990 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:37.593959 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fa9d95-csxcd" Oct 30 21:40:37.594424 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:37.594001 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fa9d95-csxcd" event={"ID":"ba20dae7-d119-48cb-9f68-f8619ebd303b","Type":"ContainerDied","Data":"012730ea1b758da898ab4103e60efe03fef9ca3d79511eebfe2a50dbecb8ba81"} Oct 30 21:40:37.594424 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:37.594046 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="012730ea1b758da898ab4103e60efe03fef9ca3d79511eebfe2a50dbecb8ba81" Oct 30 21:40:37.732005 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:37.731977 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d8ac1a-6jgw5" Oct 30 21:40:37.761016 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:37.760987 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2092c607-pw5m4"] Oct 30 21:40:37.761591 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:37.761572 4560 cpu_manager.go:414] "containerMap: removing container" podUID="ba20dae7-d119-48cb-9f68-f8619ebd303b" containerName="revoke-keys" Oct 30 21:40:37.761638 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:37.761595 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba20dae7-d119-48cb-9f68-f8619ebd303b" containerName="revoke-keys" Oct 30 21:40:37.761638 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:37.761610 4560 cpu_manager.go:414] "containerMap: removing container" podUID="a42988c9-a7e6-4dde-88d9-73ea4586dd79" containerName="revoke-keys" Oct 30 21:40:37.761638 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:37.761619 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a42988c9-a7e6-4dde-88d9-73ea4586dd79" containerName="revoke-keys" Oct 30 21:40:37.761747 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:37.761735 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba20dae7-d119-48cb-9f68-f8619ebd303b" containerName="revoke-keys" Oct 30 21:40:37.761779 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:37.761751 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="a42988c9-a7e6-4dde-88d9-73ea4586dd79" containerName="revoke-keys" Oct 30 21:40:37.764933 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:37.764916 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2092c607-pw5m4" Oct 30 21:40:37.819301 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:37.819269 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2092c607-pw5m4"] Oct 30 21:40:37.885476 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:37.885401 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a42988c9-a7e6-4dde-88d9-73ea4586dd79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a42988c9-a7e6-4dde-88d9-73ea4586dd79-maas-api-openshift-ca\") pod \"a42988c9-a7e6-4dde-88d9-73ea4586dd79\" (UID: \"a42988c9-a7e6-4dde-88d9-73ea4586dd79\") " Oct 30 21:40:37.885476 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:37.885440 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a42988c9-a7e6-4dde-88d9-73ea4586dd79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a42988c9-a7e6-4dde-88d9-73ea4586dd79-maas-api-xks-ca\") pod \"a42988c9-a7e6-4dde-88d9-73ea4586dd79\" (UID: \"a42988c9-a7e6-4dde-88d9-73ea4586dd79\") " Oct 30 21:40:37.885685 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:37.885511 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a42988c9-a7e6-4dde-88d9-73ea4586dd79-kube-api-access-cjtrq\" (UniqueName: \"kubernetes.io/projected/a42988c9-a7e6-4dde-88d9-73ea4586dd79-kube-api-access-cjtrq\") pod \"a42988c9-a7e6-4dde-88d9-73ea4586dd79\" (UID: \"a42988c9-a7e6-4dde-88d9-73ea4586dd79\") " Oct 30 21:40:37.885685 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:37.885648 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58aea94-9df8-4efc-9ce3-bcaddf98826c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2092c607-pw5m4\" (UID: \"f58aea94-9df8-4efc-9ce3-bcaddf98826c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2092c607-pw5m4" Oct 30 21:40:37.885685 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:37.885675 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktb68\" (UniqueName: \"kubernetes.io/projected/f58aea94-9df8-4efc-9ce3-bcaddf98826c-kube-api-access-ktb68\") pod \"maas-api-revoke-sub-2092c607-pw5m4\" (UID: \"f58aea94-9df8-4efc-9ce3-bcaddf98826c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2092c607-pw5m4" Oct 30 21:40:37.885842 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:37.885697 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58aea94-9df8-4efc-9ce3-bcaddf98826c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2092c607-pw5m4\" (UID: \"f58aea94-9df8-4efc-9ce3-bcaddf98826c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2092c607-pw5m4" Oct 30 21:40:37.885842 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:37.885810 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a42988c9-a7e6-4dde-88d9-73ea4586dd79-maas-api-openshift-ca" pod "a42988c9-a7e6-4dde-88d9-73ea4586dd79" (UID: "a42988c9-a7e6-4dde-88d9-73ea4586dd79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:37.887598 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:37.887574 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a42988c9-a7e6-4dde-88d9-73ea4586dd79-maas-api-xks-ca" pod "a42988c9-a7e6-4dde-88d9-73ea4586dd79" (UID: "a42988c9-a7e6-4dde-88d9-73ea4586dd79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:37.887770 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:37.887750 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a42988c9-a7e6-4dde-88d9-73ea4586dd79-kube-api-access-cjtrq" pod "a42988c9-a7e6-4dde-88d9-73ea4586dd79" (UID: "a42988c9-a7e6-4dde-88d9-73ea4586dd79"). InnerVolumeSpecName "kube-api-access-cjtrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:37.986940 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:37.986887 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58aea94-9df8-4efc-9ce3-bcaddf98826c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2092c607-pw5m4\" (UID: \"f58aea94-9df8-4efc-9ce3-bcaddf98826c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2092c607-pw5m4" Oct 30 21:40:37.987099 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:37.987077 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58aea94-9df8-4efc-9ce3-bcaddf98826c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2092c607-pw5m4\" (UID: \"f58aea94-9df8-4efc-9ce3-bcaddf98826c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2092c607-pw5m4" Oct 30 21:40:37.987166 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:37.987120 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktb68\" (UniqueName: \"kubernetes.io/projected/f58aea94-9df8-4efc-9ce3-bcaddf98826c-kube-api-access-ktb68\") pod \"maas-api-revoke-sub-2092c607-pw5m4\" (UID: \"f58aea94-9df8-4efc-9ce3-bcaddf98826c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2092c607-pw5m4" Oct 30 21:40:37.987212 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:37.987180 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a42988c9-a7e6-4dde-88d9-73ea4586dd79-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:37.987212 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:37.987198 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a42988c9-a7e6-4dde-88d9-73ea4586dd79-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:37.987320 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:37.987215 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjtrq\" (UniqueName: \"kubernetes.io/projected/a42988c9-a7e6-4dde-88d9-73ea4586dd79-kube-api-access-cjtrq\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:37.987809 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:37.987784 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58aea94-9df8-4efc-9ce3-bcaddf98826c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2092c607-pw5m4\" (UID: \"f58aea94-9df8-4efc-9ce3-bcaddf98826c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2092c607-pw5m4" Oct 30 21:40:37.989267 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:37.989248 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58aea94-9df8-4efc-9ce3-bcaddf98826c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2092c607-pw5m4\" (UID: \"f58aea94-9df8-4efc-9ce3-bcaddf98826c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2092c607-pw5m4" Oct 30 21:40:38.044727 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:38.044706 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktb68\" (UniqueName: \"kubernetes.io/projected/f58aea94-9df8-4efc-9ce3-bcaddf98826c-kube-api-access-ktb68\") pod \"maas-api-revoke-sub-2092c607-pw5m4\" (UID: \"f58aea94-9df8-4efc-9ce3-bcaddf98826c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2092c607-pw5m4" Oct 30 21:40:38.077530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:38.077507 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2092c607-pw5m4" Oct 30 21:40:38.209071 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:38.208995 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pcphw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:40:38.209071 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:38.209057 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" podUID="34448c97-8c89-4c69-9e3b-8f513d88b803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 21:40:38.315941 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:38.315904 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:40:38.450906 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:38.450874 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2092c607-pw5m4"] Oct 30 21:40:38.459407 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:40:38.459298 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58aea94_9df8_4efc_9ce3_bcaddf98826c.slice/crio-2b3c1f45aab8df31761bf8e778bb20ba58a18d457eac7bd09a6a80269626f894 WatchSource:0}: Error finding container 2b3c1f45aab8df31761bf8e778bb20ba58a18d457eac7bd09a6a80269626f894: Status 404 returned error can't find the container with id 2b3c1f45aab8df31761bf8e778bb20ba58a18d457eac7bd09a6a80269626f894 Oct 30 21:40:38.601353 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:38.601310 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2092c607-pw5m4" event={"ID":"f58aea94-9df8-4efc-9ce3-bcaddf98826c","Type":"ContainerStarted","Data":"2b3c1f45aab8df31761bf8e778bb20ba58a18d457eac7bd09a6a80269626f894"} Oct 30 21:40:38.602873 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:38.602844 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d8ac1a-6jgw5" event={"ID":"a42988c9-a7e6-4dde-88d9-73ea4586dd79","Type":"ContainerDied","Data":"db3a277e002444a19405fdda0d3820791d60161841669e4872ed666eebf950c3"} Oct 30 21:40:38.602978 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:38.602886 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d8ac1a-6jgw5" Oct 30 21:40:38.602978 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:38.602896 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db3a277e002444a19405fdda0d3820791d60161841669e4872ed666eebf950c3" Oct 30 21:40:39.251364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:39.250641 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" Oct 30 21:40:39.251364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:39.251071 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" Oct 30 21:40:39.251867 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:39.251847 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" Oct 30 21:40:39.610198 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:39.610143 4560 generic.go:356] "Generic (PLEG): container finished" podID="f58aea94-9df8-4efc-9ce3-bcaddf98826c" containerID="94565846875b3db65cf8357d9a823cf23a4f6a4a2abbaa94d8b10c30a3651f17" exitCode=0 Oct 30 21:40:39.610644 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:39.610228 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2092c607-pw5m4" event={"ID":"f58aea94-9df8-4efc-9ce3-bcaddf98826c","Type":"ContainerDied","Data":"94565846875b3db65cf8357d9a823cf23a4f6a4a2abbaa94d8b10c30a3651f17"} Oct 30 21:40:39.781425 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:39.780582 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" Oct 30 21:40:39.801111 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:39.801076 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" Oct 30 21:40:40.632529 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:40.632502 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" Oct 30 21:40:40.767215 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:40.767186 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2092c607-pw5m4" Oct 30 21:40:40.826625 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:40.826595 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f58aea94-9df8-4efc-9ce3-bcaddf98826c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58aea94-9df8-4efc-9ce3-bcaddf98826c-maas-api-xks-ca\") pod \"f58aea94-9df8-4efc-9ce3-bcaddf98826c\" (UID: \"f58aea94-9df8-4efc-9ce3-bcaddf98826c\") " Oct 30 21:40:40.826821 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:40.826662 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f58aea94-9df8-4efc-9ce3-bcaddf98826c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58aea94-9df8-4efc-9ce3-bcaddf98826c-maas-api-openshift-ca\") pod \"f58aea94-9df8-4efc-9ce3-bcaddf98826c\" (UID: \"f58aea94-9df8-4efc-9ce3-bcaddf98826c\") " Oct 30 21:40:40.826821 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:40.826701 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f58aea94-9df8-4efc-9ce3-bcaddf98826c-kube-api-access-ktb68\" (UniqueName: \"kubernetes.io/projected/f58aea94-9df8-4efc-9ce3-bcaddf98826c-kube-api-access-ktb68\") pod \"f58aea94-9df8-4efc-9ce3-bcaddf98826c\" (UID: \"f58aea94-9df8-4efc-9ce3-bcaddf98826c\") " Oct 30 21:40:40.827083 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:40.827056 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f58aea94-9df8-4efc-9ce3-bcaddf98826c-maas-api-openshift-ca" pod "f58aea94-9df8-4efc-9ce3-bcaddf98826c" (UID: "f58aea94-9df8-4efc-9ce3-bcaddf98826c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:40.829386 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:40.829354 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f58aea94-9df8-4efc-9ce3-bcaddf98826c-maas-api-xks-ca" pod "f58aea94-9df8-4efc-9ce3-bcaddf98826c" (UID: "f58aea94-9df8-4efc-9ce3-bcaddf98826c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:40.831206 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:40.831179 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58aea94-9df8-4efc-9ce3-bcaddf98826c-kube-api-access-ktb68" pod "f58aea94-9df8-4efc-9ce3-bcaddf98826c" (UID: "f58aea94-9df8-4efc-9ce3-bcaddf98826c"). InnerVolumeSpecName "kube-api-access-ktb68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:40.927835 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:40.927755 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58aea94-9df8-4efc-9ce3-bcaddf98826c-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:40.927835 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:40.927796 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58aea94-9df8-4efc-9ce3-bcaddf98826c-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:40.927835 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:40.927813 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktb68\" (UniqueName: \"kubernetes.io/projected/f58aea94-9df8-4efc-9ce3-bcaddf98826c-kube-api-access-ktb68\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:41.622101 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:41.622076 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2092c607-pw5m4" Oct 30 21:40:41.622273 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:41.622111 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2092c607-pw5m4" event={"ID":"f58aea94-9df8-4efc-9ce3-bcaddf98826c","Type":"ContainerDied","Data":"2b3c1f45aab8df31761bf8e778bb20ba58a18d457eac7bd09a6a80269626f894"} Oct 30 21:40:41.622273 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:41.622161 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b3c1f45aab8df31761bf8e778bb20ba58a18d457eac7bd09a6a80269626f894" Oct 30 21:40:42.009262 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:42.009228 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df14ffda-gmsd2"] Oct 30 21:40:42.009696 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:42.009627 4560 cpu_manager.go:414] "containerMap: removing container" podUID="f58aea94-9df8-4efc-9ce3-bcaddf98826c" containerName="revoke-keys" Oct 30 21:40:42.009696 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:42.009639 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f58aea94-9df8-4efc-9ce3-bcaddf98826c" containerName="revoke-keys" Oct 30 21:40:42.009792 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:42.009706 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="f58aea94-9df8-4efc-9ce3-bcaddf98826c" containerName="revoke-keys" Oct 30 21:40:42.012984 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:42.012963 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df14ffda-gmsd2" Oct 30 21:40:42.038851 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:42.038822 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfba7e24-87b2-4396-991f-4f38aed49369-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df14ffda-gmsd2\" (UID: \"bfba7e24-87b2-4396-991f-4f38aed49369\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df14ffda-gmsd2" Oct 30 21:40:42.038991 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:42.038881 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfba7e24-87b2-4396-991f-4f38aed49369-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df14ffda-gmsd2\" (UID: \"bfba7e24-87b2-4396-991f-4f38aed49369\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df14ffda-gmsd2" Oct 30 21:40:42.038991 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:42.038959 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvqj7\" (UniqueName: \"kubernetes.io/projected/bfba7e24-87b2-4396-991f-4f38aed49369-kube-api-access-kvqj7\") pod \"maas-api-revoke-sub-df14ffda-gmsd2\" (UID: \"bfba7e24-87b2-4396-991f-4f38aed49369\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df14ffda-gmsd2" Oct 30 21:40:42.064612 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:42.064587 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:40:42.068391 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:42.068366 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df14ffda-gmsd2"] Oct 30 21:40:42.110841 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:42.110817 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:40:42.110992 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:42.110840 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:40:42.110992 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:42.110877 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:40:42.139564 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:42.139540 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfba7e24-87b2-4396-991f-4f38aed49369-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df14ffda-gmsd2\" (UID: \"bfba7e24-87b2-4396-991f-4f38aed49369\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df14ffda-gmsd2" Oct 30 21:40:42.139706 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:42.139582 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvqj7\" (UniqueName: \"kubernetes.io/projected/bfba7e24-87b2-4396-991f-4f38aed49369-kube-api-access-kvqj7\") pod \"maas-api-revoke-sub-df14ffda-gmsd2\" (UID: \"bfba7e24-87b2-4396-991f-4f38aed49369\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df14ffda-gmsd2" Oct 30 21:40:42.139706 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:42.139643 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfba7e24-87b2-4396-991f-4f38aed49369-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df14ffda-gmsd2\" (UID: \"bfba7e24-87b2-4396-991f-4f38aed49369\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df14ffda-gmsd2" Oct 30 21:40:42.140243 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:42.140225 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfba7e24-87b2-4396-991f-4f38aed49369-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df14ffda-gmsd2\" (UID: \"bfba7e24-87b2-4396-991f-4f38aed49369\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df14ffda-gmsd2" Oct 30 21:40:42.141873 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:42.141846 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfba7e24-87b2-4396-991f-4f38aed49369-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df14ffda-gmsd2\" (UID: \"bfba7e24-87b2-4396-991f-4f38aed49369\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df14ffda-gmsd2" Oct 30 21:40:42.197278 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:42.197245 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvqj7\" (UniqueName: \"kubernetes.io/projected/bfba7e24-87b2-4396-991f-4f38aed49369-kube-api-access-kvqj7\") pod \"maas-api-revoke-sub-df14ffda-gmsd2\" (UID: \"bfba7e24-87b2-4396-991f-4f38aed49369\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df14ffda-gmsd2" Oct 30 21:40:42.325213 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:42.325137 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df14ffda-gmsd2" Oct 30 21:40:42.704517 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:42.704435 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df14ffda-gmsd2"] Oct 30 21:40:42.710421 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:40:42.710396 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfba7e24_87b2_4396_991f_4f38aed49369.slice/crio-f671d8f5dcc634ac5a3a814141ddee3267501596c1c7af12b966db7f57602143 WatchSource:0}: Error finding container f671d8f5dcc634ac5a3a814141ddee3267501596c1c7af12b966db7f57602143: Status 404 returned error can't find the container with id f671d8f5dcc634ac5a3a814141ddee3267501596c1c7af12b966db7f57602143 Oct 30 21:40:43.633969 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:43.633939 4560 generic.go:356] "Generic (PLEG): container finished" podID="bfba7e24-87b2-4396-991f-4f38aed49369" containerID="77d05270c42c708d17bb96b3f8eae241fa655ef1103be12c586f0034b8183460" exitCode=0 Oct 30 21:40:43.634322 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:43.633998 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df14ffda-gmsd2" event={"ID":"bfba7e24-87b2-4396-991f-4f38aed49369","Type":"ContainerDied","Data":"77d05270c42c708d17bb96b3f8eae241fa655ef1103be12c586f0034b8183460"} Oct 30 21:40:43.634322 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:43.634083 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df14ffda-gmsd2" event={"ID":"bfba7e24-87b2-4396-991f-4f38aed49369","Type":"ContainerStarted","Data":"f671d8f5dcc634ac5a3a814141ddee3267501596c1c7af12b966db7f57602143"} Oct 30 21:40:44.777495 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:44.777465 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df14ffda-gmsd2" Oct 30 21:40:44.866832 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:44.866795 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfba7e24-87b2-4396-991f-4f38aed49369-kube-api-access-kvqj7\" (UniqueName: \"kubernetes.io/projected/bfba7e24-87b2-4396-991f-4f38aed49369-kube-api-access-kvqj7\") pod \"bfba7e24-87b2-4396-991f-4f38aed49369\" (UID: \"bfba7e24-87b2-4396-991f-4f38aed49369\") " Oct 30 21:40:44.867042 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:44.866940 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfba7e24-87b2-4396-991f-4f38aed49369-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfba7e24-87b2-4396-991f-4f38aed49369-maas-api-xks-ca\") pod \"bfba7e24-87b2-4396-991f-4f38aed49369\" (UID: \"bfba7e24-87b2-4396-991f-4f38aed49369\") " Oct 30 21:40:44.867042 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:44.866971 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfba7e24-87b2-4396-991f-4f38aed49369-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfba7e24-87b2-4396-991f-4f38aed49369-maas-api-openshift-ca\") pod \"bfba7e24-87b2-4396-991f-4f38aed49369\" (UID: \"bfba7e24-87b2-4396-991f-4f38aed49369\") " Oct 30 21:40:44.867401 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:44.867373 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfba7e24-87b2-4396-991f-4f38aed49369-maas-api-openshift-ca" pod "bfba7e24-87b2-4396-991f-4f38aed49369" (UID: "bfba7e24-87b2-4396-991f-4f38aed49369"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:44.869068 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:44.869040 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfba7e24-87b2-4396-991f-4f38aed49369-maas-api-xks-ca" pod "bfba7e24-87b2-4396-991f-4f38aed49369" (UID: "bfba7e24-87b2-4396-991f-4f38aed49369"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:44.869146 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:44.869099 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfba7e24-87b2-4396-991f-4f38aed49369-kube-api-access-kvqj7" pod "bfba7e24-87b2-4396-991f-4f38aed49369" (UID: "bfba7e24-87b2-4396-991f-4f38aed49369"). InnerVolumeSpecName "kube-api-access-kvqj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:44.936287 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:44.936206 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg"] Oct 30 21:40:44.936601 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:44.936588 4560 cpu_manager.go:414] "containerMap: removing container" podUID="bfba7e24-87b2-4396-991f-4f38aed49369" containerName="revoke-keys" Oct 30 21:40:44.936659 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:44.936603 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfba7e24-87b2-4396-991f-4f38aed49369" containerName="revoke-keys" Oct 30 21:40:44.936706 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:44.936697 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfba7e24-87b2-4396-991f-4f38aed49369" containerName="revoke-keys" Oct 30 21:40:44.939793 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:44.939777 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" Oct 30 21:40:44.968364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:44.968315 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-ssqzg\" (UID: \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" Oct 30 21:40:44.968574 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:44.968374 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-ssqzg\" (UID: \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" Oct 30 21:40:44.968574 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:44.968459 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-ssqzg\" (UID: \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" Oct 30 21:40:44.968574 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:44.968506 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-ssqzg\" (UID: \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" Oct 30 21:40:44.968706 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:44.968597 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-ssqzg\" (UID: \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" Oct 30 21:40:44.968706 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:44.968637 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-ssqzg\" (UID: \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" Oct 30 21:40:44.968772 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:44.968706 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxw95\" (UniqueName: \"kubernetes.io/projected/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-kube-api-access-xxw95\") pod \"maas-default-gateway-openshift-default-67bf986f99-ssqzg\" (UID: \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" Oct 30 21:40:44.968772 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:44.968742 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-ssqzg\" (UID: \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" Oct 30 21:40:44.968772 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:44.968768 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-ssqzg\" (UID: \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" Oct 30 21:40:44.968887 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:44.968834 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfba7e24-87b2-4396-991f-4f38aed49369-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:44.968887 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:44.968844 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfba7e24-87b2-4396-991f-4f38aed49369-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:44.968953 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:44.968885 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvqj7\" (UniqueName: \"kubernetes.io/projected/bfba7e24-87b2-4396-991f-4f38aed49369-kube-api-access-kvqj7\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:44.998755 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:44.998719 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg"] Oct 30 21:40:45.069752 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:45.069712 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-ssqzg\" (UID: \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" Oct 30 21:40:45.069922 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:45.069778 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-ssqzg\" (UID: \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" Oct 30 21:40:45.069922 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:45.069875 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-ssqzg\" (UID: \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" Oct 30 21:40:45.069922 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:45.069908 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-ssqzg\" (UID: \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" Oct 30 21:40:45.070109 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:45.069932 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-ssqzg\" (UID: \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" Oct 30 21:40:45.070109 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:45.069965 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-ssqzg\" (UID: \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" Oct 30 21:40:45.070109 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:45.070002 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-ssqzg\" (UID: \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" Oct 30 21:40:45.070109 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:45.070047 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-ssqzg\" (UID: \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" Oct 30 21:40:45.070109 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:45.070095 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxw95\" (UniqueName: \"kubernetes.io/projected/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-kube-api-access-xxw95\") pod \"maas-default-gateway-openshift-default-67bf986f99-ssqzg\" (UID: \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" Oct 30 21:40:45.070393 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:45.070363 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-ssqzg\" (UID: \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" Oct 30 21:40:45.070486 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:45.070455 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-ssqzg\" (UID: \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" Oct 30 21:40:45.070693 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:45.070669 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-ssqzg\" (UID: \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" Oct 30 21:40:45.070832 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:45.070705 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-ssqzg\" (UID: \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" Oct 30 21:40:45.071025 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:45.071003 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-ssqzg\" (UID: \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" Oct 30 21:40:45.072427 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:45.072406 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-ssqzg\" (UID: \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" Oct 30 21:40:45.072576 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:45.072557 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-ssqzg\" (UID: \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" Oct 30 21:40:45.127757 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:45.127719 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-ssqzg\" (UID: \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" Oct 30 21:40:45.127904 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:45.127862 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxw95\" (UniqueName: \"kubernetes.io/projected/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-kube-api-access-xxw95\") pod \"maas-default-gateway-openshift-default-67bf986f99-ssqzg\" (UID: \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" Oct 30 21:40:45.253259 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:45.253227 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" Oct 30 21:40:45.645889 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:45.645806 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df14ffda-gmsd2" event={"ID":"bfba7e24-87b2-4396-991f-4f38aed49369","Type":"ContainerDied","Data":"f671d8f5dcc634ac5a3a814141ddee3267501596c1c7af12b966db7f57602143"} Oct 30 21:40:45.645889 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:45.645823 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df14ffda-gmsd2" Oct 30 21:40:45.645889 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:45.645863 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f671d8f5dcc634ac5a3a814141ddee3267501596c1c7af12b966db7f57602143" Oct 30 21:40:45.684063 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:45.684017 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg"] Oct 30 21:40:45.688645 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:40:45.688617 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeda3d2f4_4c1d_4d24_8fd8_96ba3dbc93c9.slice/crio-61c2aec7212625da0613a4dca309ebae1c6fd7ff6567c25eb2e67bc9d603f422 WatchSource:0}: Error finding container 61c2aec7212625da0613a4dca309ebae1c6fd7ff6567c25eb2e67bc9d603f422: Status 404 returned error can't find the container with id 61c2aec7212625da0613a4dca309ebae1c6fd7ff6567c25eb2e67bc9d603f422 Oct 30 21:40:45.690991 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:45.690953 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:40:45.691078 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:45.691023 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:40:45.691078 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:45.691058 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:40:46.658828 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:46.658787 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" event={"ID":"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9","Type":"ContainerStarted","Data":"cb461b982e91e25e228af1a6e6bdbfc04b08a2860948353f8379df2c87c524aa"} Oct 30 21:40:46.659209 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:46.658852 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" event={"ID":"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9","Type":"ContainerStarted","Data":"61c2aec7212625da0613a4dca309ebae1c6fd7ff6567c25eb2e67bc9d603f422"} Oct 30 21:40:46.774308 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:46.774248 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" podStartSLOduration=2.774233273 podStartE2EDuration="2.774233273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:46.772115053 +0000 UTC m=+1622.326814830" watchObservedRunningTime="2026-10-30 21:40:46.774233273 +0000 UTC m=+1622.328933047" Oct 30 21:40:47.253543 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:47.253506 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" Oct 30 21:40:47.255167 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:47.255142 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-ssqzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 21:40:47.255214 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:47.255195 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" podUID="eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 21:40:48.254486 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:48.254442 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-ssqzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 21:40:48.254890 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:48.254517 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" podUID="eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 21:40:48.508729 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:48.508650 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78166b53-wjpvh"] Oct 30 21:40:48.512400 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:48.512375 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78166b53-wjpvh" Oct 30 21:40:48.564207 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:48.564174 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:40:48.564437 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:48.564201 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:40:48.564437 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:48.564218 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:40:48.605318 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:48.605279 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50c05596-08fd-4153-9de2-473ac0db79f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78166b53-wjpvh\" (UID: \"50c05596-08fd-4153-9de2-473ac0db79f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78166b53-wjpvh" Oct 30 21:40:48.605657 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:48.605627 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50c05596-08fd-4153-9de2-473ac0db79f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78166b53-wjpvh\" (UID: \"50c05596-08fd-4153-9de2-473ac0db79f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78166b53-wjpvh" Oct 30 21:40:48.605922 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:48.605901 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9z4l\" (UniqueName: \"kubernetes.io/projected/50c05596-08fd-4153-9de2-473ac0db79f2-kube-api-access-v9z4l\") pod \"maas-api-revoke-sub-78166b53-wjpvh\" (UID: \"50c05596-08fd-4153-9de2-473ac0db79f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78166b53-wjpvh" Oct 30 21:40:48.610646 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:48.610629 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:40:48.613271 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:48.613252 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78166b53-wjpvh"] Oct 30 21:40:48.706503 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:48.706471 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9z4l\" (UniqueName: \"kubernetes.io/projected/50c05596-08fd-4153-9de2-473ac0db79f2-kube-api-access-v9z4l\") pod \"maas-api-revoke-sub-78166b53-wjpvh\" (UID: \"50c05596-08fd-4153-9de2-473ac0db79f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78166b53-wjpvh" Oct 30 21:40:48.706673 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:48.706514 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50c05596-08fd-4153-9de2-473ac0db79f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78166b53-wjpvh\" (UID: \"50c05596-08fd-4153-9de2-473ac0db79f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78166b53-wjpvh" Oct 30 21:40:48.706673 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:48.706628 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50c05596-08fd-4153-9de2-473ac0db79f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78166b53-wjpvh\" (UID: \"50c05596-08fd-4153-9de2-473ac0db79f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78166b53-wjpvh" Oct 30 21:40:48.707419 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:48.707394 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50c05596-08fd-4153-9de2-473ac0db79f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78166b53-wjpvh\" (UID: \"50c05596-08fd-4153-9de2-473ac0db79f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78166b53-wjpvh" Oct 30 21:40:48.709240 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:48.709217 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50c05596-08fd-4153-9de2-473ac0db79f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78166b53-wjpvh\" (UID: \"50c05596-08fd-4153-9de2-473ac0db79f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78166b53-wjpvh" Oct 30 21:40:48.764362 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:48.764277 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9z4l\" (UniqueName: \"kubernetes.io/projected/50c05596-08fd-4153-9de2-473ac0db79f2-kube-api-access-v9z4l\") pod \"maas-api-revoke-sub-78166b53-wjpvh\" (UID: \"50c05596-08fd-4153-9de2-473ac0db79f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78166b53-wjpvh" Oct 30 21:40:48.825661 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:48.825619 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78166b53-wjpvh" Oct 30 21:40:49.251367 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:49.247589 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78166b53-wjpvh"] Oct 30 21:40:49.254571 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:49.254536 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-ssqzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 21:40:49.255012 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:49.254597 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" podUID="eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 21:40:49.257459 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:40:49.257317 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50c05596_08fd_4153_9de2_473ac0db79f2.slice/crio-29f831d198eb41bb3b59601efb59c253292a2afd07460ceda8ca0fab90f2fc44 WatchSource:0}: Error finding container 29f831d198eb41bb3b59601efb59c253292a2afd07460ceda8ca0fab90f2fc44: Status 404 returned error can't find the container with id 29f831d198eb41bb3b59601efb59c253292a2afd07460ceda8ca0fab90f2fc44 Oct 30 21:40:49.677292 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:49.677258 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78166b53-wjpvh" event={"ID":"50c05596-08fd-4153-9de2-473ac0db79f2","Type":"ContainerStarted","Data":"3eb16b08f572fc0e9a9e15340c1fd25802097a1d77c934f00deee00840b00382"} Oct 30 21:40:49.677487 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:49.677363 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78166b53-wjpvh" event={"ID":"50c05596-08fd-4153-9de2-473ac0db79f2","Type":"ContainerStarted","Data":"29f831d198eb41bb3b59601efb59c253292a2afd07460ceda8ca0fab90f2fc44"} Oct 30 21:40:49.689415 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:40:49.689376 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:40:49.789005 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:49.788950 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78166b53-wjpvh" podStartSLOduration=1.788935966 podStartE2EDuration="1.788935966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:49.786486475 +0000 UTC m=+1625.341186253" watchObservedRunningTime="2026-10-30 21:40:49.788935966 +0000 UTC m=+1625.343635744" Oct 30 21:40:50.253915 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:50.253871 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-ssqzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 21:40:50.254065 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:50.253940 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" podUID="eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 21:40:50.696547 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:50.696456 4560 generic.go:356] "Generic (PLEG): container finished" podID="50c05596-08fd-4153-9de2-473ac0db79f2" containerID="3eb16b08f572fc0e9a9e15340c1fd25802097a1d77c934f00deee00840b00382" exitCode=0 Oct 30 21:40:50.696547 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:50.696531 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78166b53-wjpvh" event={"ID":"50c05596-08fd-4153-9de2-473ac0db79f2","Type":"ContainerDied","Data":"3eb16b08f572fc0e9a9e15340c1fd25802097a1d77c934f00deee00840b00382"} Oct 30 21:40:51.215510 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:51.215469 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8647ba4f-x56gq"] Oct 30 21:40:51.219138 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:51.219115 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8647ba4f-x56gq" Oct 30 21:40:51.253711 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:51.253676 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-ssqzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 21:40:51.253855 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:51.253730 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" podUID="eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 21:40:51.281876 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:51.281840 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8647ba4f-x56gq"] Oct 30 21:40:51.361181 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:51.361147 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqsmc\" (UniqueName: \"kubernetes.io/projected/93390da6-be64-4eb2-a121-f90e7bbc4775-kube-api-access-wqsmc\") pod \"maas-api-revoke-sub-8647ba4f-x56gq\" (UID: \"93390da6-be64-4eb2-a121-f90e7bbc4775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8647ba4f-x56gq" Oct 30 21:40:51.361366 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:51.361204 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93390da6-be64-4eb2-a121-f90e7bbc4775-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8647ba4f-x56gq\" (UID: \"93390da6-be64-4eb2-a121-f90e7bbc4775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8647ba4f-x56gq" Oct 30 21:40:51.361448 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:51.361412 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93390da6-be64-4eb2-a121-f90e7bbc4775-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8647ba4f-x56gq\" (UID: \"93390da6-be64-4eb2-a121-f90e7bbc4775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8647ba4f-x56gq" Oct 30 21:40:51.462390 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:51.462335 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93390da6-be64-4eb2-a121-f90e7bbc4775-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8647ba4f-x56gq\" (UID: \"93390da6-be64-4eb2-a121-f90e7bbc4775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8647ba4f-x56gq" Oct 30 21:40:51.462594 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:51.462400 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqsmc\" (UniqueName: \"kubernetes.io/projected/93390da6-be64-4eb2-a121-f90e7bbc4775-kube-api-access-wqsmc\") pod \"maas-api-revoke-sub-8647ba4f-x56gq\" (UID: \"93390da6-be64-4eb2-a121-f90e7bbc4775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8647ba4f-x56gq" Oct 30 21:40:51.462594 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:51.462443 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93390da6-be64-4eb2-a121-f90e7bbc4775-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8647ba4f-x56gq\" (UID: \"93390da6-be64-4eb2-a121-f90e7bbc4775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8647ba4f-x56gq" Oct 30 21:40:51.463171 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:51.463143 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93390da6-be64-4eb2-a121-f90e7bbc4775-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8647ba4f-x56gq\" (UID: \"93390da6-be64-4eb2-a121-f90e7bbc4775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8647ba4f-x56gq" Oct 30 21:40:51.464741 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:51.464720 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93390da6-be64-4eb2-a121-f90e7bbc4775-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8647ba4f-x56gq\" (UID: \"93390da6-be64-4eb2-a121-f90e7bbc4775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8647ba4f-x56gq" Oct 30 21:40:51.520282 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:51.520248 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqsmc\" (UniqueName: \"kubernetes.io/projected/93390da6-be64-4eb2-a121-f90e7bbc4775-kube-api-access-wqsmc\") pod \"maas-api-revoke-sub-8647ba4f-x56gq\" (UID: \"93390da6-be64-4eb2-a121-f90e7bbc4775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8647ba4f-x56gq" Oct 30 21:40:51.554173 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:51.554133 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8647ba4f-x56gq" Oct 30 21:40:51.831021 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:51.830995 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78166b53-wjpvh" Oct 30 21:40:51.930264 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:51.930226 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8647ba4f-x56gq"] Oct 30 21:40:51.935634 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:40:51.935610 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93390da6_be64_4eb2_a121_f90e7bbc4775.slice/crio-d2c0a79a22418d58d55f8224d61f10f210bd37bdfbd066d61079369898cecd60 WatchSource:0}: Error finding container d2c0a79a22418d58d55f8224d61f10f210bd37bdfbd066d61079369898cecd60: Status 404 returned error can't find the container with id d2c0a79a22418d58d55f8224d61f10f210bd37bdfbd066d61079369898cecd60 Oct 30 21:40:51.967388 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:51.967367 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50c05596-08fd-4153-9de2-473ac0db79f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50c05596-08fd-4153-9de2-473ac0db79f2-maas-api-openshift-ca\") pod \"50c05596-08fd-4153-9de2-473ac0db79f2\" (UID: \"50c05596-08fd-4153-9de2-473ac0db79f2\") " Oct 30 21:40:51.967478 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:51.967419 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50c05596-08fd-4153-9de2-473ac0db79f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50c05596-08fd-4153-9de2-473ac0db79f2-maas-api-xks-ca\") pod \"50c05596-08fd-4153-9de2-473ac0db79f2\" (UID: \"50c05596-08fd-4153-9de2-473ac0db79f2\") " Oct 30 21:40:51.967534 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:51.967500 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50c05596-08fd-4153-9de2-473ac0db79f2-kube-api-access-v9z4l\" (UniqueName: \"kubernetes.io/projected/50c05596-08fd-4153-9de2-473ac0db79f2-kube-api-access-v9z4l\") pod \"50c05596-08fd-4153-9de2-473ac0db79f2\" (UID: \"50c05596-08fd-4153-9de2-473ac0db79f2\") " Oct 30 21:40:51.967753 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:51.967722 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50c05596-08fd-4153-9de2-473ac0db79f2-maas-api-openshift-ca" pod "50c05596-08fd-4153-9de2-473ac0db79f2" (UID: "50c05596-08fd-4153-9de2-473ac0db79f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:51.969467 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:51.969437 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50c05596-08fd-4153-9de2-473ac0db79f2-maas-api-xks-ca" pod "50c05596-08fd-4153-9de2-473ac0db79f2" (UID: "50c05596-08fd-4153-9de2-473ac0db79f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:51.969888 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:51.969837 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50c05596-08fd-4153-9de2-473ac0db79f2-kube-api-access-v9z4l" pod "50c05596-08fd-4153-9de2-473ac0db79f2" (UID: "50c05596-08fd-4153-9de2-473ac0db79f2"). InnerVolumeSpecName "kube-api-access-v9z4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:52.068856 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:52.068775 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50c05596-08fd-4153-9de2-473ac0db79f2-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:52.068856 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:52.068806 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50c05596-08fd-4153-9de2-473ac0db79f2-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:52.068856 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:52.068818 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9z4l\" (UniqueName: \"kubernetes.io/projected/50c05596-08fd-4153-9de2-473ac0db79f2-kube-api-access-v9z4l\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:52.254666 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:52.254637 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-ssqzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 21:40:52.254785 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:52.254683 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" podUID="eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 21:40:52.709245 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:52.709203 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8647ba4f-x56gq" event={"ID":"93390da6-be64-4eb2-a121-f90e7bbc4775","Type":"ContainerStarted","Data":"f9f8868087cd2f85910460949156d3ebc246eb37b05a763dd3263d1a633e39f7"} Oct 30 21:40:52.709469 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:52.709272 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8647ba4f-x56gq" event={"ID":"93390da6-be64-4eb2-a121-f90e7bbc4775","Type":"ContainerStarted","Data":"d2c0a79a22418d58d55f8224d61f10f210bd37bdfbd066d61079369898cecd60"} Oct 30 21:40:52.710655 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:52.710632 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78166b53-wjpvh" event={"ID":"50c05596-08fd-4153-9de2-473ac0db79f2","Type":"ContainerDied","Data":"29f831d198eb41bb3b59601efb59c253292a2afd07460ceda8ca0fab90f2fc44"} Oct 30 21:40:52.710745 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:52.710672 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29f831d198eb41bb3b59601efb59c253292a2afd07460ceda8ca0fab90f2fc44" Oct 30 21:40:52.710745 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:52.710642 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78166b53-wjpvh" Oct 30 21:40:52.825806 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:52.825747 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8647ba4f-x56gq" podStartSLOduration=1.82573228 podStartE2EDuration="1.82573228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:52.823861281 +0000 UTC m=+1628.378561059" watchObservedRunningTime="2026-10-30 21:40:52.82573228 +0000 UTC m=+1628.380432118" Oct 30 21:40:53.254630 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:53.254587 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-ssqzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 21:40:53.255014 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:53.254665 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" podUID="eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 21:40:53.720081 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:53.719978 4560 generic.go:356] "Generic (PLEG): container finished" podID="93390da6-be64-4eb2-a121-f90e7bbc4775" containerID="f9f8868087cd2f85910460949156d3ebc246eb37b05a763dd3263d1a633e39f7" exitCode=0 Oct 30 21:40:53.720081 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:53.720038 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8647ba4f-x56gq" event={"ID":"93390da6-be64-4eb2-a121-f90e7bbc4775","Type":"ContainerDied","Data":"f9f8868087cd2f85910460949156d3ebc246eb37b05a763dd3263d1a633e39f7"} Oct 30 21:40:54.258387 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:54.258334 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" Oct 30 21:40:54.258874 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:54.258785 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" Oct 30 21:40:54.259640 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:54.259619 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" Oct 30 21:40:54.854560 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:54.854532 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8647ba4f-x56gq" Oct 30 21:40:55.001252 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:55.001173 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93390da6-be64-4eb2-a121-f90e7bbc4775-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93390da6-be64-4eb2-a121-f90e7bbc4775-maas-api-openshift-ca\") pod \"93390da6-be64-4eb2-a121-f90e7bbc4775\" (UID: \"93390da6-be64-4eb2-a121-f90e7bbc4775\") " Oct 30 21:40:55.001252 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:55.001221 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93390da6-be64-4eb2-a121-f90e7bbc4775-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93390da6-be64-4eb2-a121-f90e7bbc4775-maas-api-xks-ca\") pod \"93390da6-be64-4eb2-a121-f90e7bbc4775\" (UID: \"93390da6-be64-4eb2-a121-f90e7bbc4775\") " Oct 30 21:40:55.001252 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:55.001248 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93390da6-be64-4eb2-a121-f90e7bbc4775-kube-api-access-wqsmc\" (UniqueName: \"kubernetes.io/projected/93390da6-be64-4eb2-a121-f90e7bbc4775-kube-api-access-wqsmc\") pod \"93390da6-be64-4eb2-a121-f90e7bbc4775\" (UID: \"93390da6-be64-4eb2-a121-f90e7bbc4775\") " Oct 30 21:40:55.001674 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:55.001644 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93390da6-be64-4eb2-a121-f90e7bbc4775-maas-api-openshift-ca" pod "93390da6-be64-4eb2-a121-f90e7bbc4775" (UID: "93390da6-be64-4eb2-a121-f90e7bbc4775"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:55.003493 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:55.003469 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93390da6-be64-4eb2-a121-f90e7bbc4775-kube-api-access-wqsmc" pod "93390da6-be64-4eb2-a121-f90e7bbc4775" (UID: "93390da6-be64-4eb2-a121-f90e7bbc4775"). InnerVolumeSpecName "kube-api-access-wqsmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:55.003854 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:55.003821 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93390da6-be64-4eb2-a121-f90e7bbc4775-maas-api-xks-ca" pod "93390da6-be64-4eb2-a121-f90e7bbc4775" (UID: "93390da6-be64-4eb2-a121-f90e7bbc4775"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:55.102840 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:55.102804 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93390da6-be64-4eb2-a121-f90e7bbc4775-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:55.102840 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:55.102836 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93390da6-be64-4eb2-a121-f90e7bbc4775-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:55.102840 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:55.102845 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqsmc\" (UniqueName: \"kubernetes.io/projected/93390da6-be64-4eb2-a121-f90e7bbc4775-kube-api-access-wqsmc\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:55.731807 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:55.731772 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8647ba4f-x56gq" event={"ID":"93390da6-be64-4eb2-a121-f90e7bbc4775","Type":"ContainerDied","Data":"d2c0a79a22418d58d55f8224d61f10f210bd37bdfbd066d61079369898cecd60"} Oct 30 21:40:55.731807 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:55.731801 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8647ba4f-x56gq" Oct 30 21:40:55.732261 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:55.731824 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2c0a79a22418d58d55f8224d61f10f210bd37bdfbd066d61079369898cecd60" Oct 30 21:40:59.342474 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:59.342437 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c9bafde-5lfrj"] Oct 30 21:40:59.342977 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:59.342958 4560 cpu_manager.go:414] "containerMap: removing container" podUID="93390da6-be64-4eb2-a121-f90e7bbc4775" containerName="revoke-keys" Oct 30 21:40:59.343020 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:59.342981 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93390da6-be64-4eb2-a121-f90e7bbc4775" containerName="revoke-keys" Oct 30 21:40:59.343020 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:59.343007 4560 cpu_manager.go:414] "containerMap: removing container" podUID="50c05596-08fd-4153-9de2-473ac0db79f2" containerName="revoke-keys" Oct 30 21:40:59.343020 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:59.343012 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50c05596-08fd-4153-9de2-473ac0db79f2" containerName="revoke-keys" Oct 30 21:40:59.343113 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:59.343083 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="93390da6-be64-4eb2-a121-f90e7bbc4775" containerName="revoke-keys" Oct 30 21:40:59.343113 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:59.343094 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="50c05596-08fd-4153-9de2-473ac0db79f2" containerName="revoke-keys" Oct 30 21:40:59.347419 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:59.347401 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9bafde-5lfrj" Oct 30 21:40:59.397687 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:59.397663 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:40:59.444184 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:59.444156 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:40:59.444313 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:59.444156 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:40:59.444313 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:59.444237 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:40:59.444526 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:59.444500 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdvqw\" (UniqueName: \"kubernetes.io/projected/5c4b67bd-02b1-4938-a169-9a08d0ac3530-kube-api-access-sdvqw\") pod \"maas-api-revoke-sub-2c9bafde-5lfrj\" (UID: \"5c4b67bd-02b1-4938-a169-9a08d0ac3530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9bafde-5lfrj" Oct 30 21:40:59.444620 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:59.444606 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c4b67bd-02b1-4938-a169-9a08d0ac3530-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c9bafde-5lfrj\" (UID: \"5c4b67bd-02b1-4938-a169-9a08d0ac3530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9bafde-5lfrj" Oct 30 21:40:59.444662 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:59.444640 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c4b67bd-02b1-4938-a169-9a08d0ac3530-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c9bafde-5lfrj\" (UID: \"5c4b67bd-02b1-4938-a169-9a08d0ac3530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9bafde-5lfrj" Oct 30 21:40:59.447149 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:59.447126 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c9bafde-5lfrj"] Oct 30 21:40:59.545896 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:59.545866 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c4b67bd-02b1-4938-a169-9a08d0ac3530-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c9bafde-5lfrj\" (UID: \"5c4b67bd-02b1-4938-a169-9a08d0ac3530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9bafde-5lfrj" Oct 30 21:40:59.546024 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:59.545906 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c4b67bd-02b1-4938-a169-9a08d0ac3530-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c9bafde-5lfrj\" (UID: \"5c4b67bd-02b1-4938-a169-9a08d0ac3530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9bafde-5lfrj" Oct 30 21:40:59.546024 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:59.545998 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdvqw\" (UniqueName: \"kubernetes.io/projected/5c4b67bd-02b1-4938-a169-9a08d0ac3530-kube-api-access-sdvqw\") pod \"maas-api-revoke-sub-2c9bafde-5lfrj\" (UID: \"5c4b67bd-02b1-4938-a169-9a08d0ac3530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9bafde-5lfrj" Oct 30 21:40:59.546630 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:59.546603 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c4b67bd-02b1-4938-a169-9a08d0ac3530-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c9bafde-5lfrj\" (UID: \"5c4b67bd-02b1-4938-a169-9a08d0ac3530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9bafde-5lfrj" Oct 30 21:40:59.548197 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:59.548174 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c4b67bd-02b1-4938-a169-9a08d0ac3530-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c9bafde-5lfrj\" (UID: \"5c4b67bd-02b1-4938-a169-9a08d0ac3530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9bafde-5lfrj" Oct 30 21:40:59.603826 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:59.603758 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdvqw\" (UniqueName: \"kubernetes.io/projected/5c4b67bd-02b1-4938-a169-9a08d0ac3530-kube-api-access-sdvqw\") pod \"maas-api-revoke-sub-2c9bafde-5lfrj\" (UID: \"5c4b67bd-02b1-4938-a169-9a08d0ac3530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9bafde-5lfrj" Oct 30 21:40:59.659542 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:40:59.659517 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9bafde-5lfrj" Oct 30 21:41:00.059806 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:00.059770 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c9bafde-5lfrj"] Oct 30 21:41:00.065867 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:41:00.065842 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c4b67bd_02b1_4938_a169_9a08d0ac3530.slice/crio-7b9d1ae37b7007e44799f8e2bd3caef6477b466674288b8d31745d18a0c69bad WatchSource:0}: Error finding container 7b9d1ae37b7007e44799f8e2bd3caef6477b466674288b8d31745d18a0c69bad: Status 404 returned error can't find the container with id 7b9d1ae37b7007e44799f8e2bd3caef6477b466674288b8d31745d18a0c69bad Oct 30 21:41:00.539933 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:00.539903 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw"] Oct 30 21:41:00.543640 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:00.543621 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" Oct 30 21:41:00.595044 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:00.594979 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-a2dced-openshift-default-dockercfg-ll74c\"" Oct 30 21:41:00.644485 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:00.644455 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw"] Oct 30 21:41:00.656241 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:00.656210 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3a33e86-072f-458f-837d-d349fb6981da-credential-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw\" (UID: \"b3a33e86-072f-458f-837d-d349fb6981da\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" Oct 30 21:41:00.656401 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:00.656274 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3a33e86-072f-458f-837d-d349fb6981da-istio-data\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw\" (UID: \"b3a33e86-072f-458f-837d-d349fb6981da\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" Oct 30 21:41:00.656401 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:00.656376 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3a33e86-072f-458f-837d-d349fb6981da-istio-token\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw\" (UID: \"b3a33e86-072f-458f-837d-d349fb6981da\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" Oct 30 21:41:00.656482 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:00.656461 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3a33e86-072f-458f-837d-d349fb6981da-workload-certs\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw\" (UID: \"b3a33e86-072f-458f-837d-d349fb6981da\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" Oct 30 21:41:00.656526 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:00.656505 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3a33e86-072f-458f-837d-d349fb6981da-istio-envoy\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw\" (UID: \"b3a33e86-072f-458f-837d-d349fb6981da\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" Oct 30 21:41:00.656613 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:00.656591 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3a33e86-072f-458f-837d-d349fb6981da-istio-podinfo\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw\" (UID: \"b3a33e86-072f-458f-837d-d349fb6981da\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" Oct 30 21:41:00.656692 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:00.656678 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3a33e86-072f-458f-837d-d349fb6981da-istiod-ca-cert\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw\" (UID: \"b3a33e86-072f-458f-837d-d349fb6981da\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" Oct 30 21:41:00.656795 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:00.656780 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-276gf\" (UniqueName: \"kubernetes.io/projected/b3a33e86-072f-458f-837d-d349fb6981da-kube-api-access-276gf\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw\" (UID: \"b3a33e86-072f-458f-837d-d349fb6981da\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" Oct 30 21:41:00.656827 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:00.656817 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3a33e86-072f-458f-837d-d349fb6981da-workload-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw\" (UID: \"b3a33e86-072f-458f-837d-d349fb6981da\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" Oct 30 21:41:00.757907 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:00.757875 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3a33e86-072f-458f-837d-d349fb6981da-workload-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw\" (UID: \"b3a33e86-072f-458f-837d-d349fb6981da\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" Oct 30 21:41:00.758053 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:00.757955 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3a33e86-072f-458f-837d-d349fb6981da-credential-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw\" (UID: \"b3a33e86-072f-458f-837d-d349fb6981da\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" Oct 30 21:41:00.758053 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:00.757987 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3a33e86-072f-458f-837d-d349fb6981da-istio-data\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw\" (UID: \"b3a33e86-072f-458f-837d-d349fb6981da\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" Oct 30 21:41:00.758053 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:00.758013 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3a33e86-072f-458f-837d-d349fb6981da-istio-token\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw\" (UID: \"b3a33e86-072f-458f-837d-d349fb6981da\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" Oct 30 21:41:00.758171 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:00.758050 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3a33e86-072f-458f-837d-d349fb6981da-workload-certs\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw\" (UID: \"b3a33e86-072f-458f-837d-d349fb6981da\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" Oct 30 21:41:00.758171 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:00.758111 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3a33e86-072f-458f-837d-d349fb6981da-istio-envoy\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw\" (UID: \"b3a33e86-072f-458f-837d-d349fb6981da\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" Oct 30 21:41:00.758245 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:00.758169 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3a33e86-072f-458f-837d-d349fb6981da-istio-podinfo\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw\" (UID: \"b3a33e86-072f-458f-837d-d349fb6981da\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" Oct 30 21:41:00.758245 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:00.758234 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3a33e86-072f-458f-837d-d349fb6981da-istiod-ca-cert\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw\" (UID: \"b3a33e86-072f-458f-837d-d349fb6981da\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" Oct 30 21:41:00.758322 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:00.758289 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3a33e86-072f-458f-837d-d349fb6981da-workload-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw\" (UID: \"b3a33e86-072f-458f-837d-d349fb6981da\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" Oct 30 21:41:00.758322 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:00.758315 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-276gf\" (UniqueName: \"kubernetes.io/projected/b3a33e86-072f-458f-837d-d349fb6981da-kube-api-access-276gf\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw\" (UID: \"b3a33e86-072f-458f-837d-d349fb6981da\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" Oct 30 21:41:00.758434 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:00.758353 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3a33e86-072f-458f-837d-d349fb6981da-credential-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw\" (UID: \"b3a33e86-072f-458f-837d-d349fb6981da\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" Oct 30 21:41:00.758434 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:00.758410 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3a33e86-072f-458f-837d-d349fb6981da-istio-data\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw\" (UID: \"b3a33e86-072f-458f-837d-d349fb6981da\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" Oct 30 21:41:00.758595 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:00.758571 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3a33e86-072f-458f-837d-d349fb6981da-workload-certs\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw\" (UID: \"b3a33e86-072f-458f-837d-d349fb6981da\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" Oct 30 21:41:00.758908 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:00.758886 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3a33e86-072f-458f-837d-d349fb6981da-istiod-ca-cert\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw\" (UID: \"b3a33e86-072f-458f-837d-d349fb6981da\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" Oct 30 21:41:00.760752 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:00.760733 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3a33e86-072f-458f-837d-d349fb6981da-istio-podinfo\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw\" (UID: \"b3a33e86-072f-458f-837d-d349fb6981da\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" Oct 30 21:41:00.760982 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:00.760960 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3a33e86-072f-458f-837d-d349fb6981da-istio-envoy\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw\" (UID: \"b3a33e86-072f-458f-837d-d349fb6981da\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" Oct 30 21:41:00.762860 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:00.762835 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9bafde-5lfrj" event={"ID":"5c4b67bd-02b1-4938-a169-9a08d0ac3530","Type":"ContainerStarted","Data":"7462a7ee02990109b9793d64bed33580a6a1082f6f9128a072b4f71f2529922f"} Oct 30 21:41:00.762981 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:00.762890 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9bafde-5lfrj" event={"ID":"5c4b67bd-02b1-4938-a169-9a08d0ac3530","Type":"ContainerStarted","Data":"7b9d1ae37b7007e44799f8e2bd3caef6477b466674288b8d31745d18a0c69bad"} Oct 30 21:41:00.815565 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:00.815541 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3a33e86-072f-458f-837d-d349fb6981da-istio-token\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw\" (UID: \"b3a33e86-072f-458f-837d-d349fb6981da\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" Oct 30 21:41:00.815769 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:00.815749 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-276gf\" (UniqueName: \"kubernetes.io/projected/b3a33e86-072f-458f-837d-d349fb6981da-kube-api-access-276gf\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw\" (UID: \"b3a33e86-072f-458f-837d-d349fb6981da\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" Oct 30 21:41:00.865379 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:00.865302 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" Oct 30 21:41:00.873472 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:00.873430 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9bafde-5lfrj" podStartSLOduration=1.873415582 podStartE2EDuration="1.873415582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:00.871213854 +0000 UTC m=+1636.425913631" watchObservedRunningTime="2026-10-30 21:41:00.873415582 +0000 UTC m=+1636.428115359" Oct 30 21:41:01.294305 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:01.294270 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw"] Oct 30 21:41:01.299566 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:41:01.299537 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3a33e86_072f_458f_837d_d349fb6981da.slice/crio-3d4f4e2864c04ae563d54834613fdd209da8583cc675424725fdde4acdd660f8 WatchSource:0}: Error finding container 3d4f4e2864c04ae563d54834613fdd209da8583cc675424725fdde4acdd660f8: Status 404 returned error can't find the container with id 3d4f4e2864c04ae563d54834613fdd209da8583cc675424725fdde4acdd660f8 Oct 30 21:41:01.301948 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:01.301910 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:41:01.302058 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:01.301993 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:41:01.302058 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:01.302041 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:41:01.774179 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:01.774136 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" event={"ID":"b3a33e86-072f-458f-837d-d349fb6981da","Type":"ContainerStarted","Data":"053382ed30d17e0dafc5129ff6221cd1b0f790b6ecad7af1c52b219cef88b5e2"} Oct 30 21:41:01.774792 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:01.774204 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" event={"ID":"b3a33e86-072f-458f-837d-d349fb6981da","Type":"ContainerStarted","Data":"3d4f4e2864c04ae563d54834613fdd209da8583cc675424725fdde4acdd660f8"} Oct 30 21:41:01.777007 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:01.776954 4560 generic.go:356] "Generic (PLEG): container finished" podID="5c4b67bd-02b1-4938-a169-9a08d0ac3530" containerID="7462a7ee02990109b9793d64bed33580a6a1082f6f9128a072b4f71f2529922f" exitCode=0 Oct 30 21:41:01.777149 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:01.777063 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9bafde-5lfrj" event={"ID":"5c4b67bd-02b1-4938-a169-9a08d0ac3530","Type":"ContainerDied","Data":"7462a7ee02990109b9793d64bed33580a6a1082f6f9128a072b4f71f2529922f"} Oct 30 21:41:01.819867 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:01.819772 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw"] Oct 30 21:41:01.831915 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:01.831879 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" Oct 30 21:41:01.867807 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:01.866436 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" Oct 30 21:41:01.880054 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:01.880006 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw"] Oct 30 21:41:01.883400 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:01.883297 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-991335-openshift-default-dockercfg-xsdjt\"" Oct 30 21:41:01.991435 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:01.989877 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d83608f-bcb0-4aab-b14e-29846561a3d3-istio-token\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw\" (UID: \"5d83608f-bcb0-4aab-b14e-29846561a3d3\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" Oct 30 21:41:01.991435 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:01.989918 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d83608f-bcb0-4aab-b14e-29846561a3d3-credential-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw\" (UID: \"5d83608f-bcb0-4aab-b14e-29846561a3d3\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" Oct 30 21:41:01.991435 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:01.989953 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d83608f-bcb0-4aab-b14e-29846561a3d3-istio-data\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw\" (UID: \"5d83608f-bcb0-4aab-b14e-29846561a3d3\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" Oct 30 21:41:01.991435 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:01.989977 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d83608f-bcb0-4aab-b14e-29846561a3d3-istio-podinfo\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw\" (UID: \"5d83608f-bcb0-4aab-b14e-29846561a3d3\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" Oct 30 21:41:01.991435 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:01.990002 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkpsq\" (UniqueName: \"kubernetes.io/projected/5d83608f-bcb0-4aab-b14e-29846561a3d3-kube-api-access-nkpsq\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw\" (UID: \"5d83608f-bcb0-4aab-b14e-29846561a3d3\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" Oct 30 21:41:01.991435 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:01.990047 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d83608f-bcb0-4aab-b14e-29846561a3d3-istiod-ca-cert\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw\" (UID: \"5d83608f-bcb0-4aab-b14e-29846561a3d3\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" Oct 30 21:41:01.991435 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:01.990087 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d83608f-bcb0-4aab-b14e-29846561a3d3-istio-envoy\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw\" (UID: \"5d83608f-bcb0-4aab-b14e-29846561a3d3\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" Oct 30 21:41:01.991435 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:01.990104 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d83608f-bcb0-4aab-b14e-29846561a3d3-workload-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw\" (UID: \"5d83608f-bcb0-4aab-b14e-29846561a3d3\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" Oct 30 21:41:01.991435 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:01.990119 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d83608f-bcb0-4aab-b14e-29846561a3d3-workload-certs\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw\" (UID: \"5d83608f-bcb0-4aab-b14e-29846561a3d3\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" Oct 30 21:41:02.084694 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:02.084579 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" podStartSLOduration=2.084561177 podStartE2EDuration="2.084561177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:01.974587663 +0000 UTC m=+1637.529287441" watchObservedRunningTime="2026-10-30 21:41:02.084561177 +0000 UTC m=+1637.639260954" Oct 30 21:41:02.090785 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:02.090759 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d83608f-bcb0-4aab-b14e-29846561a3d3-istio-token\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw\" (UID: \"5d83608f-bcb0-4aab-b14e-29846561a3d3\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" Oct 30 21:41:02.090923 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:02.090796 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d83608f-bcb0-4aab-b14e-29846561a3d3-credential-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw\" (UID: \"5d83608f-bcb0-4aab-b14e-29846561a3d3\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" Oct 30 21:41:02.090923 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:02.090829 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d83608f-bcb0-4aab-b14e-29846561a3d3-istio-data\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw\" (UID: \"5d83608f-bcb0-4aab-b14e-29846561a3d3\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" Oct 30 21:41:02.091103 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:02.090970 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d83608f-bcb0-4aab-b14e-29846561a3d3-istio-podinfo\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw\" (UID: \"5d83608f-bcb0-4aab-b14e-29846561a3d3\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" Oct 30 21:41:02.091103 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:02.091015 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkpsq\" (UniqueName: \"kubernetes.io/projected/5d83608f-bcb0-4aab-b14e-29846561a3d3-kube-api-access-nkpsq\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw\" (UID: \"5d83608f-bcb0-4aab-b14e-29846561a3d3\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" Oct 30 21:41:02.091199 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:02.091099 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d83608f-bcb0-4aab-b14e-29846561a3d3-istiod-ca-cert\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw\" (UID: \"5d83608f-bcb0-4aab-b14e-29846561a3d3\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" Oct 30 21:41:02.091199 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:02.091172 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d83608f-bcb0-4aab-b14e-29846561a3d3-istio-envoy\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw\" (UID: \"5d83608f-bcb0-4aab-b14e-29846561a3d3\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" Oct 30 21:41:02.091310 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:02.091201 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d83608f-bcb0-4aab-b14e-29846561a3d3-workload-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw\" (UID: \"5d83608f-bcb0-4aab-b14e-29846561a3d3\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" Oct 30 21:41:02.091310 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:02.091216 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d83608f-bcb0-4aab-b14e-29846561a3d3-istio-data\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw\" (UID: \"5d83608f-bcb0-4aab-b14e-29846561a3d3\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" Oct 30 21:41:02.091310 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:02.091225 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d83608f-bcb0-4aab-b14e-29846561a3d3-workload-certs\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw\" (UID: \"5d83608f-bcb0-4aab-b14e-29846561a3d3\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" Oct 30 21:41:02.091310 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:02.091270 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d83608f-bcb0-4aab-b14e-29846561a3d3-credential-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw\" (UID: \"5d83608f-bcb0-4aab-b14e-29846561a3d3\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" Oct 30 21:41:02.091558 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:02.091535 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d83608f-bcb0-4aab-b14e-29846561a3d3-workload-certs\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw\" (UID: \"5d83608f-bcb0-4aab-b14e-29846561a3d3\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" Oct 30 21:41:02.091720 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:02.091694 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d83608f-bcb0-4aab-b14e-29846561a3d3-workload-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw\" (UID: \"5d83608f-bcb0-4aab-b14e-29846561a3d3\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" Oct 30 21:41:02.092113 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:02.092092 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d83608f-bcb0-4aab-b14e-29846561a3d3-istiod-ca-cert\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw\" (UID: \"5d83608f-bcb0-4aab-b14e-29846561a3d3\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" Oct 30 21:41:02.093705 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:02.093674 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d83608f-bcb0-4aab-b14e-29846561a3d3-istio-envoy\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw\" (UID: \"5d83608f-bcb0-4aab-b14e-29846561a3d3\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" Oct 30 21:41:02.093786 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:02.093722 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d83608f-bcb0-4aab-b14e-29846561a3d3-istio-podinfo\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw\" (UID: \"5d83608f-bcb0-4aab-b14e-29846561a3d3\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" Oct 30 21:41:02.156361 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:02.156302 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkpsq\" (UniqueName: \"kubernetes.io/projected/5d83608f-bcb0-4aab-b14e-29846561a3d3-kube-api-access-nkpsq\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw\" (UID: \"5d83608f-bcb0-4aab-b14e-29846561a3d3\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" Oct 30 21:41:02.157793 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:02.157760 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d83608f-bcb0-4aab-b14e-29846561a3d3-istio-token\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw\" (UID: \"5d83608f-bcb0-4aab-b14e-29846561a3d3\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" Oct 30 21:41:02.178003 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:02.176873 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" Oct 30 21:41:02.691368 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:02.690872 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw"] Oct 30 21:41:02.699911 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:02.699842 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:41:02.700135 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:02.700110 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:41:02.700282 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:02.700261 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:41:02.797409 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:02.784988 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" event={"ID":"5d83608f-bcb0-4aab-b14e-29846561a3d3","Type":"ContainerStarted","Data":"4e68d60430e1cf08f68b979f6ad3f14b97e062866971fc892c9a3d296cc4c7a8"} Oct 30 21:41:02.867567 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:02.867440 4560 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:41:02.867567 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:02.867495 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" podUID="b3a33e86-072f-458f-837d-d349fb6981da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 21:41:03.122126 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:03.122095 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9bafde-5lfrj" Oct 30 21:41:03.298727 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:03.298570 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c4b67bd-02b1-4938-a169-9a08d0ac3530-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c4b67bd-02b1-4938-a169-9a08d0ac3530-maas-api-openshift-ca\") pod \"5c4b67bd-02b1-4938-a169-9a08d0ac3530\" (UID: \"5c4b67bd-02b1-4938-a169-9a08d0ac3530\") " Oct 30 21:41:03.299024 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:03.299000 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c4b67bd-02b1-4938-a169-9a08d0ac3530-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c4b67bd-02b1-4938-a169-9a08d0ac3530-maas-api-xks-ca\") pod \"5c4b67bd-02b1-4938-a169-9a08d0ac3530\" (UID: \"5c4b67bd-02b1-4938-a169-9a08d0ac3530\") " Oct 30 21:41:03.310375 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:03.299215 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c4b67bd-02b1-4938-a169-9a08d0ac3530-kube-api-access-sdvqw\" (UniqueName: \"kubernetes.io/projected/5c4b67bd-02b1-4938-a169-9a08d0ac3530-kube-api-access-sdvqw\") pod \"5c4b67bd-02b1-4938-a169-9a08d0ac3530\" (UID: \"5c4b67bd-02b1-4938-a169-9a08d0ac3530\") " Oct 30 21:41:03.318366 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:03.316897 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c4b67bd-02b1-4938-a169-9a08d0ac3530-maas-api-openshift-ca" pod "5c4b67bd-02b1-4938-a169-9a08d0ac3530" (UID: "5c4b67bd-02b1-4938-a169-9a08d0ac3530"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:03.319121 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:03.319095 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c4b67bd-02b1-4938-a169-9a08d0ac3530-maas-api-xks-ca" pod "5c4b67bd-02b1-4938-a169-9a08d0ac3530" (UID: "5c4b67bd-02b1-4938-a169-9a08d0ac3530"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:03.321177 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:03.320967 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c4b67bd-02b1-4938-a169-9a08d0ac3530-kube-api-access-sdvqw" pod "5c4b67bd-02b1-4938-a169-9a08d0ac3530" (UID: "5c4b67bd-02b1-4938-a169-9a08d0ac3530"). InnerVolumeSpecName "kube-api-access-sdvqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:03.411898 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:03.411826 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c4b67bd-02b1-4938-a169-9a08d0ac3530-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:03.411898 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:03.411853 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c4b67bd-02b1-4938-a169-9a08d0ac3530-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:03.411898 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:03.411862 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdvqw\" (UniqueName: \"kubernetes.io/projected/5c4b67bd-02b1-4938-a169-9a08d0ac3530-kube-api-access-sdvqw\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:03.792071 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:03.792028 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" event={"ID":"5d83608f-bcb0-4aab-b14e-29846561a3d3","Type":"ContainerStarted","Data":"eec07909c0ab2a065e177d21c7cae12a71d37a6aba1c35630a172459f2a204d8"} Oct 30 21:41:03.793362 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:03.793320 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9bafde-5lfrj" Oct 30 21:41:03.793515 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:03.793333 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9bafde-5lfrj" event={"ID":"5c4b67bd-02b1-4938-a169-9a08d0ac3530","Type":"ContainerDied","Data":"7b9d1ae37b7007e44799f8e2bd3caef6477b466674288b8d31745d18a0c69bad"} Oct 30 21:41:03.793515 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:03.793409 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b9d1ae37b7007e44799f8e2bd3caef6477b466674288b8d31745d18a0c69bad" Oct 30 21:41:03.865874 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:03.865831 4560 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:41:03.866493 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:03.866460 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" podUID="b3a33e86-072f-458f-837d-d349fb6981da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 21:41:03.911566 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:03.911504 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" podStartSLOduration=2.911489794 podStartE2EDuration="2.911489794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:03.909029065 +0000 UTC m=+1639.463728844" watchObservedRunningTime="2026-10-30 21:41:03.911489794 +0000 UTC m=+1639.466189629" Oct 30 21:41:04.181028 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:04.180935 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" Oct 30 21:41:04.183542 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:04.183515 4560 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:41:04.183733 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:04.183714 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" podUID="5d83608f-bcb0-4aab-b14e-29846561a3d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 21:41:04.880323 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:04.878878 4560 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:41:04.880323 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:04.878951 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" podUID="b3a33e86-072f-458f-837d-d349fb6981da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 21:41:05.178474 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:05.178378 4560 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:41:05.178743 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:05.178689 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" podUID="5d83608f-bcb0-4aab-b14e-29846561a3d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 21:41:05.866362 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:05.865705 4560 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:41:05.866362 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:05.865767 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" podUID="b3a33e86-072f-458f-837d-d349fb6981da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 21:41:06.177944 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:06.177846 4560 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:41:06.177944 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:06.177908 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" podUID="5d83608f-bcb0-4aab-b14e-29846561a3d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 21:41:06.875822 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:06.875784 4560 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:41:06.876483 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:06.876442 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" podUID="b3a33e86-072f-458f-837d-d349fb6981da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 21:41:06.981408 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:06.981376 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cbb50c8-pbs2v"] Oct 30 21:41:06.981953 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:06.981938 4560 cpu_manager.go:414] "containerMap: removing container" podUID="5c4b67bd-02b1-4938-a169-9a08d0ac3530" containerName="revoke-keys" Oct 30 21:41:06.982044 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:06.982034 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c4b67bd-02b1-4938-a169-9a08d0ac3530" containerName="revoke-keys" Oct 30 21:41:06.982211 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:06.982200 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c4b67bd-02b1-4938-a169-9a08d0ac3530" containerName="revoke-keys" Oct 30 21:41:06.985328 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:06.985312 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cbb50c8-pbs2v" Oct 30 21:41:07.058364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:07.055714 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:41:07.058364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:07.055879 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:41:07.082435 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:07.082400 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cbb50c8-pbs2v"] Oct 30 21:41:07.090038 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:07.090015 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b0280a3-57f9-4fdd-9e32-3c796db6f2d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cbb50c8-pbs2v\" (UID: \"5b0280a3-57f9-4fdd-9e32-3c796db6f2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cbb50c8-pbs2v" Oct 30 21:41:07.090264 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:07.090249 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b0280a3-57f9-4fdd-9e32-3c796db6f2d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cbb50c8-pbs2v\" (UID: \"5b0280a3-57f9-4fdd-9e32-3c796db6f2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cbb50c8-pbs2v" Oct 30 21:41:07.090370 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:07.090354 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tscj\" (UniqueName: \"kubernetes.io/projected/5b0280a3-57f9-4fdd-9e32-3c796db6f2d1-kube-api-access-8tscj\") pod \"maas-api-revoke-sub-5cbb50c8-pbs2v\" (UID: \"5b0280a3-57f9-4fdd-9e32-3c796db6f2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cbb50c8-pbs2v" Oct 30 21:41:07.176233 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:07.175691 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:41:07.176233 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:07.175794 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:41:07.177610 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:07.177581 4560 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:41:07.177818 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:07.177791 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" podUID="5d83608f-bcb0-4aab-b14e-29846561a3d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 21:41:07.191216 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:07.191190 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b0280a3-57f9-4fdd-9e32-3c796db6f2d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cbb50c8-pbs2v\" (UID: \"5b0280a3-57f9-4fdd-9e32-3c796db6f2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cbb50c8-pbs2v" Oct 30 21:41:07.191603 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:07.191282 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b0280a3-57f9-4fdd-9e32-3c796db6f2d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cbb50c8-pbs2v\" (UID: \"5b0280a3-57f9-4fdd-9e32-3c796db6f2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cbb50c8-pbs2v" Oct 30 21:41:07.191603 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:07.191320 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tscj\" (UniqueName: \"kubernetes.io/projected/5b0280a3-57f9-4fdd-9e32-3c796db6f2d1-kube-api-access-8tscj\") pod \"maas-api-revoke-sub-5cbb50c8-pbs2v\" (UID: \"5b0280a3-57f9-4fdd-9e32-3c796db6f2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cbb50c8-pbs2v" Oct 30 21:41:07.191997 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:07.191978 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b0280a3-57f9-4fdd-9e32-3c796db6f2d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cbb50c8-pbs2v\" (UID: \"5b0280a3-57f9-4fdd-9e32-3c796db6f2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cbb50c8-pbs2v" Oct 30 21:41:07.193675 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:07.193657 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b0280a3-57f9-4fdd-9e32-3c796db6f2d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cbb50c8-pbs2v\" (UID: \"5b0280a3-57f9-4fdd-9e32-3c796db6f2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cbb50c8-pbs2v" Oct 30 21:41:07.262200 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:07.262168 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tscj\" (UniqueName: \"kubernetes.io/projected/5b0280a3-57f9-4fdd-9e32-3c796db6f2d1-kube-api-access-8tscj\") pod \"maas-api-revoke-sub-5cbb50c8-pbs2v\" (UID: \"5b0280a3-57f9-4fdd-9e32-3c796db6f2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cbb50c8-pbs2v" Oct 30 21:41:07.299080 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:07.299049 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cbb50c8-pbs2v" Oct 30 21:41:07.701355 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:07.701283 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cbb50c8-pbs2v"] Oct 30 21:41:07.710482 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:41:07.710450 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0280a3_57f9_4fdd_9e32_3c796db6f2d1.slice/crio-5344a1e284b792a76fb00f225f72aefa75e6118eab47583798bf170822a9a9aa WatchSource:0}: Error finding container 5344a1e284b792a76fb00f225f72aefa75e6118eab47583798bf170822a9a9aa: Status 404 returned error can't find the container with id 5344a1e284b792a76fb00f225f72aefa75e6118eab47583798bf170822a9a9aa Oct 30 21:41:07.817363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:07.817319 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cbb50c8-pbs2v" event={"ID":"5b0280a3-57f9-4fdd-9e32-3c796db6f2d1","Type":"ContainerStarted","Data":"5344a1e284b792a76fb00f225f72aefa75e6118eab47583798bf170822a9a9aa"} Oct 30 21:41:07.866709 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:07.866676 4560 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:41:07.866859 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:07.866741 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" podUID="b3a33e86-072f-458f-837d-d349fb6981da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 21:41:08.177928 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:08.177888 4560 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:41:08.178126 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:08.177957 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" podUID="5d83608f-bcb0-4aab-b14e-29846561a3d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 21:41:08.846291 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:08.846134 4560 generic.go:356] "Generic (PLEG): container finished" podID="5b0280a3-57f9-4fdd-9e32-3c796db6f2d1" containerID="e207886496bf336adc77ce2b8d8fae1216f98e383e89af316c7c0b0d05cd8ad3" exitCode=0 Oct 30 21:41:08.846291 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:08.846290 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cbb50c8-pbs2v" event={"ID":"5b0280a3-57f9-4fdd-9e32-3c796db6f2d1","Type":"ContainerDied","Data":"e207886496bf336adc77ce2b8d8fae1216f98e383e89af316c7c0b0d05cd8ad3"} Oct 30 21:41:09.177544 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:09.177462 4560 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:41:09.177544 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:09.177529 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" podUID="5d83608f-bcb0-4aab-b14e-29846561a3d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 21:41:09.403502 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:09.403466 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" Oct 30 21:41:09.403938 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:09.403909 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" Oct 30 21:41:09.404849 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:09.404823 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" Oct 30 21:41:09.992791 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:09.992766 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cbb50c8-pbs2v" Oct 30 21:41:10.022147 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:10.022104 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b0280a3-57f9-4fdd-9e32-3c796db6f2d1-kube-api-access-8tscj\" (UniqueName: \"kubernetes.io/projected/5b0280a3-57f9-4fdd-9e32-3c796db6f2d1-kube-api-access-8tscj\") pod \"5b0280a3-57f9-4fdd-9e32-3c796db6f2d1\" (UID: \"5b0280a3-57f9-4fdd-9e32-3c796db6f2d1\") " Oct 30 21:41:10.022434 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:10.022317 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b0280a3-57f9-4fdd-9e32-3c796db6f2d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b0280a3-57f9-4fdd-9e32-3c796db6f2d1-maas-api-openshift-ca\") pod \"5b0280a3-57f9-4fdd-9e32-3c796db6f2d1\" (UID: \"5b0280a3-57f9-4fdd-9e32-3c796db6f2d1\") " Oct 30 21:41:10.022514 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:10.022458 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b0280a3-57f9-4fdd-9e32-3c796db6f2d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b0280a3-57f9-4fdd-9e32-3c796db6f2d1-maas-api-xks-ca\") pod \"5b0280a3-57f9-4fdd-9e32-3c796db6f2d1\" (UID: \"5b0280a3-57f9-4fdd-9e32-3c796db6f2d1\") " Oct 30 21:41:10.024225 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:10.024086 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b0280a3-57f9-4fdd-9e32-3c796db6f2d1-maas-api-openshift-ca" pod "5b0280a3-57f9-4fdd-9e32-3c796db6f2d1" (UID: "5b0280a3-57f9-4fdd-9e32-3c796db6f2d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:10.028863 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:10.028807 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b0280a3-57f9-4fdd-9e32-3c796db6f2d1-maas-api-xks-ca" pod "5b0280a3-57f9-4fdd-9e32-3c796db6f2d1" (UID: "5b0280a3-57f9-4fdd-9e32-3c796db6f2d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:10.033502 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:10.033455 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b0280a3-57f9-4fdd-9e32-3c796db6f2d1-kube-api-access-8tscj" pod "5b0280a3-57f9-4fdd-9e32-3c796db6f2d1" (UID: "5b0280a3-57f9-4fdd-9e32-3c796db6f2d1"). InnerVolumeSpecName "kube-api-access-8tscj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:10.123986 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:10.123883 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b0280a3-57f9-4fdd-9e32-3c796db6f2d1-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:10.123986 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:10.123926 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tscj\" (UniqueName: \"kubernetes.io/projected/5b0280a3-57f9-4fdd-9e32-3c796db6f2d1-kube-api-access-8tscj\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:10.123986 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:10.123942 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b0280a3-57f9-4fdd-9e32-3c796db6f2d1-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:10.182321 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:10.182291 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" Oct 30 21:41:10.182613 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:10.182582 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" Oct 30 21:41:10.183230 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:10.183211 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" Oct 30 21:41:10.859057 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:10.858989 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cbb50c8-pbs2v" Oct 30 21:41:10.859057 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:10.859020 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cbb50c8-pbs2v" event={"ID":"5b0280a3-57f9-4fdd-9e32-3c796db6f2d1","Type":"ContainerDied","Data":"5344a1e284b792a76fb00f225f72aefa75e6118eab47583798bf170822a9a9aa"} Oct 30 21:41:10.859235 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:10.859068 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5344a1e284b792a76fb00f225f72aefa75e6118eab47583798bf170822a9a9aa" Oct 30 21:41:19.688724 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:41:19.688680 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:41:22.112357 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:22.112315 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5"] Oct 30 21:41:22.112773 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:22.112726 4560 cpu_manager.go:414] "containerMap: removing container" podUID="5b0280a3-57f9-4fdd-9e32-3c796db6f2d1" containerName="revoke-keys" Oct 30 21:41:22.112773 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:22.112738 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b0280a3-57f9-4fdd-9e32-3c796db6f2d1" containerName="revoke-keys" Oct 30 21:41:22.112856 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:22.112821 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b0280a3-57f9-4fdd-9e32-3c796db6f2d1" containerName="revoke-keys" Oct 30 21:41:22.117724 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:22.117701 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" Oct 30 21:41:22.218752 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:22.218709 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-19dd4f\"/\"distinct-w1-19dd4f-kserve-self-signed-certs\"" Oct 30 21:41:22.221979 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:22.221955 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5"] Oct 30 21:41:22.259418 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:22.259383 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e974bae1-bb18-46b0-a4ee-4c01d6455e32-model-cache\") pod \"distinct-w1-19dd4f-kserve-798f8b8b56-pxct5\" (UID: \"e974bae1-bb18-46b0-a4ee-4c01d6455e32\") " pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" Oct 30 21:41:22.259573 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:22.259434 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e974bae1-bb18-46b0-a4ee-4c01d6455e32-home\") pod \"distinct-w1-19dd4f-kserve-798f8b8b56-pxct5\" (UID: \"e974bae1-bb18-46b0-a4ee-4c01d6455e32\") " pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" Oct 30 21:41:22.259573 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:22.259524 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq6bk\" (UniqueName: \"kubernetes.io/projected/e974bae1-bb18-46b0-a4ee-4c01d6455e32-kube-api-access-lq6bk\") pod \"distinct-w1-19dd4f-kserve-798f8b8b56-pxct5\" (UID: \"e974bae1-bb18-46b0-a4ee-4c01d6455e32\") " pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" Oct 30 21:41:22.259644 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:22.259620 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e974bae1-bb18-46b0-a4ee-4c01d6455e32-dshm\") pod \"distinct-w1-19dd4f-kserve-798f8b8b56-pxct5\" (UID: \"e974bae1-bb18-46b0-a4ee-4c01d6455e32\") " pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" Oct 30 21:41:22.259678 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:22.259645 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e974bae1-bb18-46b0-a4ee-4c01d6455e32-tls-certs\") pod \"distinct-w1-19dd4f-kserve-798f8b8b56-pxct5\" (UID: \"e974bae1-bb18-46b0-a4ee-4c01d6455e32\") " pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" Oct 30 21:41:22.259678 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:22.259668 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e974bae1-bb18-46b0-a4ee-4c01d6455e32-tmp-dir\") pod \"distinct-w1-19dd4f-kserve-798f8b8b56-pxct5\" (UID: \"e974bae1-bb18-46b0-a4ee-4c01d6455e32\") " pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" Oct 30 21:41:22.360261 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:22.360228 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e974bae1-bb18-46b0-a4ee-4c01d6455e32-tls-certs\") pod \"distinct-w1-19dd4f-kserve-798f8b8b56-pxct5\" (UID: \"e974bae1-bb18-46b0-a4ee-4c01d6455e32\") " pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" Oct 30 21:41:22.360490 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:22.360269 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e974bae1-bb18-46b0-a4ee-4c01d6455e32-tmp-dir\") pod \"distinct-w1-19dd4f-kserve-798f8b8b56-pxct5\" (UID: \"e974bae1-bb18-46b0-a4ee-4c01d6455e32\") " pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" Oct 30 21:41:22.360490 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:22.360327 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e974bae1-bb18-46b0-a4ee-4c01d6455e32-model-cache\") pod \"distinct-w1-19dd4f-kserve-798f8b8b56-pxct5\" (UID: \"e974bae1-bb18-46b0-a4ee-4c01d6455e32\") " pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" Oct 30 21:41:22.360490 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:22.360400 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e974bae1-bb18-46b0-a4ee-4c01d6455e32-home\") pod \"distinct-w1-19dd4f-kserve-798f8b8b56-pxct5\" (UID: \"e974bae1-bb18-46b0-a4ee-4c01d6455e32\") " pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" Oct 30 21:41:22.360490 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:22.360429 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq6bk\" (UniqueName: \"kubernetes.io/projected/e974bae1-bb18-46b0-a4ee-4c01d6455e32-kube-api-access-lq6bk\") pod \"distinct-w1-19dd4f-kserve-798f8b8b56-pxct5\" (UID: \"e974bae1-bb18-46b0-a4ee-4c01d6455e32\") " pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" Oct 30 21:41:22.360715 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:22.360511 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e974bae1-bb18-46b0-a4ee-4c01d6455e32-dshm\") pod \"distinct-w1-19dd4f-kserve-798f8b8b56-pxct5\" (UID: \"e974bae1-bb18-46b0-a4ee-4c01d6455e32\") " pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" Oct 30 21:41:22.360913 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:22.360878 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e974bae1-bb18-46b0-a4ee-4c01d6455e32-tmp-dir\") pod \"distinct-w1-19dd4f-kserve-798f8b8b56-pxct5\" (UID: \"e974bae1-bb18-46b0-a4ee-4c01d6455e32\") " pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" Oct 30 21:41:22.360913 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:22.360894 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e974bae1-bb18-46b0-a4ee-4c01d6455e32-model-cache\") pod \"distinct-w1-19dd4f-kserve-798f8b8b56-pxct5\" (UID: \"e974bae1-bb18-46b0-a4ee-4c01d6455e32\") " pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" Oct 30 21:41:22.361104 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:22.360939 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e974bae1-bb18-46b0-a4ee-4c01d6455e32-home\") pod \"distinct-w1-19dd4f-kserve-798f8b8b56-pxct5\" (UID: \"e974bae1-bb18-46b0-a4ee-4c01d6455e32\") " pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" Oct 30 21:41:22.362727 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:22.362673 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e974bae1-bb18-46b0-a4ee-4c01d6455e32-dshm\") pod \"distinct-w1-19dd4f-kserve-798f8b8b56-pxct5\" (UID: \"e974bae1-bb18-46b0-a4ee-4c01d6455e32\") " pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" Oct 30 21:41:22.362833 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:22.362816 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e974bae1-bb18-46b0-a4ee-4c01d6455e32-tls-certs\") pod \"distinct-w1-19dd4f-kserve-798f8b8b56-pxct5\" (UID: \"e974bae1-bb18-46b0-a4ee-4c01d6455e32\") " pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" Oct 30 21:41:22.419016 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:22.418977 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq6bk\" (UniqueName: \"kubernetes.io/projected/e974bae1-bb18-46b0-a4ee-4c01d6455e32-kube-api-access-lq6bk\") pod \"distinct-w1-19dd4f-kserve-798f8b8b56-pxct5\" (UID: \"e974bae1-bb18-46b0-a4ee-4c01d6455e32\") " pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" Oct 30 21:41:22.430692 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:22.430658 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" Oct 30 21:41:22.861220 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:22.861181 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5"] Oct 30 21:41:22.866638 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:41:22.866612 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode974bae1_bb18_46b0_a4ee_4c01d6455e32.slice/crio-81bd16177b46aa77a2a102dffc6948e89db134f3307ef8df47a23dc713536613 WatchSource:0}: Error finding container 81bd16177b46aa77a2a102dffc6948e89db134f3307ef8df47a23dc713536613: Status 404 returned error can't find the container with id 81bd16177b46aa77a2a102dffc6948e89db134f3307ef8df47a23dc713536613 Oct 30 21:41:22.928505 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:22.928471 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" event={"ID":"e974bae1-bb18-46b0-a4ee-4c01d6455e32","Type":"ContainerStarted","Data":"9ba844e55ec42cf1177ef19a3bb2a883454f05d4434cfbaed552f2d7922486cc"} Oct 30 21:41:22.928640 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:22.928523 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" event={"ID":"e974bae1-bb18-46b0-a4ee-4c01d6455e32","Type":"ContainerStarted","Data":"81bd16177b46aa77a2a102dffc6948e89db134f3307ef8df47a23dc713536613"} Oct 30 21:41:23.040631 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:23.040574 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" podStartSLOduration=1.040557521 podStartE2EDuration="1.040557521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:23.038227928 +0000 UTC m=+1658.592927709" watchObservedRunningTime="2026-10-30 21:41:23.040557521 +0000 UTC m=+1658.595257299" Oct 30 21:41:23.431224 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:23.431186 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" Oct 30 21:41:25.368326 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:25.368280 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:41:30.474605 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:30.474511 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2"] Oct 30 21:41:30.478682 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:30.478654 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" Oct 30 21:41:30.527666 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:30.527629 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-ed76b8-openshift-default-dockercfg-47s56\"" Oct 30 21:41:30.587222 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:30.587178 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5e17746-9f03-4857-b551-bc5625cd5b67-istio-data\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2\" (UID: \"a5e17746-9f03-4857-b551-bc5625cd5b67\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" Oct 30 21:41:30.587503 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:30.587259 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5e17746-9f03-4857-b551-bc5625cd5b67-istiod-ca-cert\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2\" (UID: \"a5e17746-9f03-4857-b551-bc5625cd5b67\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" Oct 30 21:41:30.587503 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:30.587324 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5e17746-9f03-4857-b551-bc5625cd5b67-istio-envoy\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2\" (UID: \"a5e17746-9f03-4857-b551-bc5625cd5b67\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" Oct 30 21:41:30.587503 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:30.587385 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xhfp\" (UniqueName: \"kubernetes.io/projected/a5e17746-9f03-4857-b551-bc5625cd5b67-kube-api-access-5xhfp\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2\" (UID: \"a5e17746-9f03-4857-b551-bc5625cd5b67\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" Oct 30 21:41:30.587503 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:30.587419 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e17746-9f03-4857-b551-bc5625cd5b67-credential-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2\" (UID: \"a5e17746-9f03-4857-b551-bc5625cd5b67\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" Oct 30 21:41:30.587930 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:30.587507 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5e17746-9f03-4857-b551-bc5625cd5b67-istio-podinfo\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2\" (UID: \"a5e17746-9f03-4857-b551-bc5625cd5b67\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" Oct 30 21:41:30.587930 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:30.587545 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5e17746-9f03-4857-b551-bc5625cd5b67-istio-token\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2\" (UID: \"a5e17746-9f03-4857-b551-bc5625cd5b67\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" Oct 30 21:41:30.587930 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:30.587574 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5e17746-9f03-4857-b551-bc5625cd5b67-workload-certs\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2\" (UID: \"a5e17746-9f03-4857-b551-bc5625cd5b67\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" Oct 30 21:41:30.587930 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:30.587615 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e17746-9f03-4857-b551-bc5625cd5b67-workload-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2\" (UID: \"a5e17746-9f03-4857-b551-bc5625cd5b67\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" Oct 30 21:41:30.595642 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:30.595616 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2"] Oct 30 21:41:30.688681 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:30.688648 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xhfp\" (UniqueName: \"kubernetes.io/projected/a5e17746-9f03-4857-b551-bc5625cd5b67-kube-api-access-5xhfp\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2\" (UID: \"a5e17746-9f03-4857-b551-bc5625cd5b67\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" Oct 30 21:41:30.688867 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:30.688694 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e17746-9f03-4857-b551-bc5625cd5b67-credential-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2\" (UID: \"a5e17746-9f03-4857-b551-bc5625cd5b67\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" Oct 30 21:41:30.688867 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:30.688780 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5e17746-9f03-4857-b551-bc5625cd5b67-istio-podinfo\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2\" (UID: \"a5e17746-9f03-4857-b551-bc5625cd5b67\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" Oct 30 21:41:30.688867 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:30.688815 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5e17746-9f03-4857-b551-bc5625cd5b67-istio-token\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2\" (UID: \"a5e17746-9f03-4857-b551-bc5625cd5b67\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" Oct 30 21:41:30.688867 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:30.688839 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5e17746-9f03-4857-b551-bc5625cd5b67-workload-certs\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2\" (UID: \"a5e17746-9f03-4857-b551-bc5625cd5b67\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" Oct 30 21:41:30.689147 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:30.688876 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e17746-9f03-4857-b551-bc5625cd5b67-workload-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2\" (UID: \"a5e17746-9f03-4857-b551-bc5625cd5b67\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" Oct 30 21:41:30.689147 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:30.688919 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5e17746-9f03-4857-b551-bc5625cd5b67-istio-data\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2\" (UID: \"a5e17746-9f03-4857-b551-bc5625cd5b67\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" Oct 30 21:41:30.689147 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:30.688967 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5e17746-9f03-4857-b551-bc5625cd5b67-istiod-ca-cert\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2\" (UID: \"a5e17746-9f03-4857-b551-bc5625cd5b67\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" Oct 30 21:41:30.689147 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:30.689053 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5e17746-9f03-4857-b551-bc5625cd5b67-istio-envoy\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2\" (UID: \"a5e17746-9f03-4857-b551-bc5625cd5b67\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" Oct 30 21:41:30.689374 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:30.689232 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e17746-9f03-4857-b551-bc5625cd5b67-credential-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2\" (UID: \"a5e17746-9f03-4857-b551-bc5625cd5b67\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" Oct 30 21:41:30.689374 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:30.689269 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e17746-9f03-4857-b551-bc5625cd5b67-workload-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2\" (UID: \"a5e17746-9f03-4857-b551-bc5625cd5b67\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" Oct 30 21:41:30.689374 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:30.689293 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5e17746-9f03-4857-b551-bc5625cd5b67-workload-certs\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2\" (UID: \"a5e17746-9f03-4857-b551-bc5625cd5b67\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" Oct 30 21:41:30.689545 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:30.689494 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5e17746-9f03-4857-b551-bc5625cd5b67-istio-data\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2\" (UID: \"a5e17746-9f03-4857-b551-bc5625cd5b67\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" Oct 30 21:41:30.689863 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:30.689838 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5e17746-9f03-4857-b551-bc5625cd5b67-istiod-ca-cert\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2\" (UID: \"a5e17746-9f03-4857-b551-bc5625cd5b67\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" Oct 30 21:41:30.691312 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:30.691288 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5e17746-9f03-4857-b551-bc5625cd5b67-istio-envoy\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2\" (UID: \"a5e17746-9f03-4857-b551-bc5625cd5b67\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" Oct 30 21:41:30.691543 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:30.691521 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5e17746-9f03-4857-b551-bc5625cd5b67-istio-podinfo\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2\" (UID: \"a5e17746-9f03-4857-b551-bc5625cd5b67\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" Oct 30 21:41:30.746256 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:30.746228 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5e17746-9f03-4857-b551-bc5625cd5b67-istio-token\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2\" (UID: \"a5e17746-9f03-4857-b551-bc5625cd5b67\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" Oct 30 21:41:30.746404 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:30.746308 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xhfp\" (UniqueName: \"kubernetes.io/projected/a5e17746-9f03-4857-b551-bc5625cd5b67-kube-api-access-5xhfp\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2\" (UID: \"a5e17746-9f03-4857-b551-bc5625cd5b67\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" Oct 30 21:41:30.792152 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:30.792126 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" Oct 30 21:41:31.228287 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:31.228257 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2"] Oct 30 21:41:31.233616 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:41:31.233574 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5e17746_9f03_4857_b551_bc5625cd5b67.slice/crio-8c6600b768b88a9c4c5a85c517cdb1c32727d5661f144645685ca69ad6ad8028 WatchSource:0}: Error finding container 8c6600b768b88a9c4c5a85c517cdb1c32727d5661f144645685ca69ad6ad8028: Status 404 returned error can't find the container with id 8c6600b768b88a9c4c5a85c517cdb1c32727d5661f144645685ca69ad6ad8028 Oct 30 21:41:31.236159 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:31.236120 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:41:31.236271 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:31.236204 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:41:31.236271 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:31.236251 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:41:31.513616 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:31.513544 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp"] Oct 30 21:41:31.517280 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:31.517264 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" Oct 30 21:41:31.567427 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:31.567394 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-afbe5b-openshift-default-dockercfg-jbdjs\"" Oct 30 21:41:31.577375 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:31.575712 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp"] Oct 30 21:41:31.600437 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:31.600409 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10eae6b-051d-4929-902f-abcd38146a15-workload-socket\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp\" (UID: \"f10eae6b-051d-4929-902f-abcd38146a15\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" Oct 30 21:41:31.600576 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:31.600452 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f10eae6b-051d-4929-902f-abcd38146a15-istio-data\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp\" (UID: \"f10eae6b-051d-4929-902f-abcd38146a15\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" Oct 30 21:41:31.600576 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:31.600539 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f10eae6b-051d-4929-902f-abcd38146a15-istiod-ca-cert\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp\" (UID: \"f10eae6b-051d-4929-902f-abcd38146a15\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" Oct 30 21:41:31.600696 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:31.600623 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f10eae6b-051d-4929-902f-abcd38146a15-istio-envoy\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp\" (UID: \"f10eae6b-051d-4929-902f-abcd38146a15\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" Oct 30 21:41:31.600696 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:31.600651 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f10eae6b-051d-4929-902f-abcd38146a15-workload-certs\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp\" (UID: \"f10eae6b-051d-4929-902f-abcd38146a15\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" Oct 30 21:41:31.600781 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:31.600712 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9plzk\" (UniqueName: \"kubernetes.io/projected/f10eae6b-051d-4929-902f-abcd38146a15-kube-api-access-9plzk\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp\" (UID: \"f10eae6b-051d-4929-902f-abcd38146a15\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" Oct 30 21:41:31.600781 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:31.600756 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f10eae6b-051d-4929-902f-abcd38146a15-istio-podinfo\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp\" (UID: \"f10eae6b-051d-4929-902f-abcd38146a15\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" Oct 30 21:41:31.600871 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:31.600802 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10eae6b-051d-4929-902f-abcd38146a15-credential-socket\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp\" (UID: \"f10eae6b-051d-4929-902f-abcd38146a15\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" Oct 30 21:41:31.600871 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:31.600819 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f10eae6b-051d-4929-902f-abcd38146a15-istio-token\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp\" (UID: \"f10eae6b-051d-4929-902f-abcd38146a15\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" Oct 30 21:41:31.702023 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:31.701989 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f10eae6b-051d-4929-902f-abcd38146a15-istio-podinfo\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp\" (UID: \"f10eae6b-051d-4929-902f-abcd38146a15\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" Oct 30 21:41:31.702190 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:31.702045 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10eae6b-051d-4929-902f-abcd38146a15-credential-socket\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp\" (UID: \"f10eae6b-051d-4929-902f-abcd38146a15\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" Oct 30 21:41:31.702190 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:31.702069 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f10eae6b-051d-4929-902f-abcd38146a15-istio-token\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp\" (UID: \"f10eae6b-051d-4929-902f-abcd38146a15\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" Oct 30 21:41:31.702190 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:31.702107 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10eae6b-051d-4929-902f-abcd38146a15-workload-socket\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp\" (UID: \"f10eae6b-051d-4929-902f-abcd38146a15\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" Oct 30 21:41:31.702190 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:31.702133 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f10eae6b-051d-4929-902f-abcd38146a15-istio-data\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp\" (UID: \"f10eae6b-051d-4929-902f-abcd38146a15\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" Oct 30 21:41:31.702190 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:31.702159 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f10eae6b-051d-4929-902f-abcd38146a15-istiod-ca-cert\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp\" (UID: \"f10eae6b-051d-4929-902f-abcd38146a15\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" Oct 30 21:41:31.702466 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:31.702229 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f10eae6b-051d-4929-902f-abcd38146a15-istio-envoy\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp\" (UID: \"f10eae6b-051d-4929-902f-abcd38146a15\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" Oct 30 21:41:31.702466 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:31.702258 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f10eae6b-051d-4929-902f-abcd38146a15-workload-certs\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp\" (UID: \"f10eae6b-051d-4929-902f-abcd38146a15\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" Oct 30 21:41:31.702466 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:31.702293 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9plzk\" (UniqueName: \"kubernetes.io/projected/f10eae6b-051d-4929-902f-abcd38146a15-kube-api-access-9plzk\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp\" (UID: \"f10eae6b-051d-4929-902f-abcd38146a15\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" Oct 30 21:41:31.702625 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:31.702547 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10eae6b-051d-4929-902f-abcd38146a15-credential-socket\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp\" (UID: \"f10eae6b-051d-4929-902f-abcd38146a15\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" Oct 30 21:41:31.702625 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:31.702567 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10eae6b-051d-4929-902f-abcd38146a15-workload-socket\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp\" (UID: \"f10eae6b-051d-4929-902f-abcd38146a15\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" Oct 30 21:41:31.702731 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:31.702711 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f10eae6b-051d-4929-902f-abcd38146a15-istio-data\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp\" (UID: \"f10eae6b-051d-4929-902f-abcd38146a15\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" Oct 30 21:41:31.702898 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:31.702879 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f10eae6b-051d-4929-902f-abcd38146a15-workload-certs\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp\" (UID: \"f10eae6b-051d-4929-902f-abcd38146a15\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" Oct 30 21:41:31.703122 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:31.703100 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f10eae6b-051d-4929-902f-abcd38146a15-istiod-ca-cert\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp\" (UID: \"f10eae6b-051d-4929-902f-abcd38146a15\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" Oct 30 21:41:31.704877 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:31.704860 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f10eae6b-051d-4929-902f-abcd38146a15-istio-podinfo\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp\" (UID: \"f10eae6b-051d-4929-902f-abcd38146a15\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" Oct 30 21:41:31.705445 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:31.705421 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f10eae6b-051d-4929-902f-abcd38146a15-istio-envoy\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp\" (UID: \"f10eae6b-051d-4929-902f-abcd38146a15\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" Oct 30 21:41:31.760374 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:31.760332 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f10eae6b-051d-4929-902f-abcd38146a15-istio-token\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp\" (UID: \"f10eae6b-051d-4929-902f-abcd38146a15\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" Oct 30 21:41:31.760529 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:31.760513 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9plzk\" (UniqueName: \"kubernetes.io/projected/f10eae6b-051d-4929-902f-abcd38146a15-kube-api-access-9plzk\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp\" (UID: \"f10eae6b-051d-4929-902f-abcd38146a15\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" Oct 30 21:41:31.839415 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:31.836537 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" Oct 30 21:41:31.988937 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:31.988899 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" event={"ID":"a5e17746-9f03-4857-b551-bc5625cd5b67","Type":"ContainerStarted","Data":"c7f5ea5b89b2db32d5df33653b252331f4a02ba56ade1ce3de01a176d4807c83"} Oct 30 21:41:31.989138 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:31.988967 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" event={"ID":"a5e17746-9f03-4857-b551-bc5625cd5b67","Type":"ContainerStarted","Data":"8c6600b768b88a9c4c5a85c517cdb1c32727d5661f144645685ca69ad6ad8028"} Oct 30 21:41:32.110000 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:32.109907 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" podStartSLOduration=2.10989224 podStartE2EDuration="2.10989224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:32.107131096 +0000 UTC m=+1667.661830871" watchObservedRunningTime="2026-10-30 21:41:32.10989224 +0000 UTC m=+1667.664592016" Oct 30 21:41:32.286518 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:32.286483 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp"] Oct 30 21:41:32.291072 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:41:32.291042 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf10eae6b_051d_4929_902f_abcd38146a15.slice/crio-69200a2eb9b7a974012168612839f9f243478da9e8ef21aab40d279da069407c WatchSource:0}: Error finding container 69200a2eb9b7a974012168612839f9f243478da9e8ef21aab40d279da069407c: Status 404 returned error can't find the container with id 69200a2eb9b7a974012168612839f9f243478da9e8ef21aab40d279da069407c Oct 30 21:41:32.293601 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:32.293566 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:41:32.293694 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:32.293636 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:41:32.293694 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:32.293687 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:41:32.439959 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:32.436697 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" Oct 30 21:41:32.475366 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:32.475196 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" Oct 30 21:41:32.792360 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:32.792306 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" Oct 30 21:41:32.793699 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:32.793674 4560 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:41:32.793813 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:32.793733 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" podUID="a5e17746-9f03-4857-b551-bc5625cd5b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 21:41:32.995914 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:32.995879 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" event={"ID":"f10eae6b-051d-4929-902f-abcd38146a15","Type":"ContainerStarted","Data":"61b1b054f2ab58b12b0dc2fb000f420582a975311487be609addeeea4e488779"} Oct 30 21:41:32.996064 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:32.995934 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" event={"ID":"f10eae6b-051d-4929-902f-abcd38146a15","Type":"ContainerStarted","Data":"69200a2eb9b7a974012168612839f9f243478da9e8ef21aab40d279da069407c"} Oct 30 21:41:33.007552 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:33.007524 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" Oct 30 21:41:33.135537 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:33.134805 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" podStartSLOduration=2.134782988 podStartE2EDuration="2.134782988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:33.123558412 +0000 UTC m=+1668.678258188" watchObservedRunningTime="2026-10-30 21:41:33.134782988 +0000 UTC m=+1668.689482765" Oct 30 21:41:33.793116 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:33.793076 4560 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:41:33.793528 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:33.793130 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" podUID="a5e17746-9f03-4857-b551-bc5625cd5b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 21:41:33.849364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:33.848427 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" Oct 30 21:41:33.863363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:33.849904 4560 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:41:33.863363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:33.849960 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" podUID="f10eae6b-051d-4929-902f-abcd38146a15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 21:41:34.793253 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:34.793209 4560 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:41:34.793791 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:34.793263 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" podUID="a5e17746-9f03-4857-b551-bc5625cd5b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 21:41:34.853371 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:34.845698 4560 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:41:34.853371 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:34.845760 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" podUID="f10eae6b-051d-4929-902f-abcd38146a15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 21:41:35.793523 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:35.793485 4560 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:41:35.793991 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:35.793547 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" podUID="a5e17746-9f03-4857-b551-bc5625cd5b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 21:41:35.846752 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:35.846713 4560 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:41:35.849150 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:35.849117 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" podUID="f10eae6b-051d-4929-902f-abcd38146a15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 21:41:36.792682 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:36.792653 4560 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:41:36.792917 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:36.792894 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" podUID="a5e17746-9f03-4857-b551-bc5625cd5b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 21:41:36.838753 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:36.838702 4560 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:41:36.839333 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:36.839299 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" podUID="f10eae6b-051d-4929-902f-abcd38146a15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 21:41:37.837833 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:37.837792 4560 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:41:37.838161 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:37.838113 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" podUID="f10eae6b-051d-4929-902f-abcd38146a15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 21:41:38.056373 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:38.055630 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" Oct 30 21:41:38.056842 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:38.056543 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" Oct 30 21:41:38.057579 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:38.057553 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" Oct 30 21:41:38.837950 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:38.837915 4560 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:41:38.838131 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:38.837968 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" podUID="f10eae6b-051d-4929-902f-abcd38146a15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 21:41:39.841242 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:39.841213 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" Oct 30 21:41:39.841656 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:39.841577 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" Oct 30 21:41:39.842167 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:41:39.842150 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" Oct 30 21:41:49.686315 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:41:49.686265 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:42:00.098722 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:00.098688 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:42:19.684935 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:42:19.684892 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:42:21.936400 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:21.936364 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg"] Oct 30 21:42:21.940058 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:21.940034 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" Oct 30 21:42:22.026949 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:22.026910 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5t4c\" (UniqueName: \"kubernetes.io/projected/1e044c02-0053-40e8-9f80-e806cd2d9ae5-kube-api-access-k5t4c\") pod \"distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg\" (UID: \"1e044c02-0053-40e8-9f80-e806cd2d9ae5\") " pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" Oct 30 21:42:22.027168 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:22.026983 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e044c02-0053-40e8-9f80-e806cd2d9ae5-tls-certs\") pod \"distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg\" (UID: \"1e044c02-0053-40e8-9f80-e806cd2d9ae5\") " pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" Oct 30 21:42:22.027168 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:22.027081 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e044c02-0053-40e8-9f80-e806cd2d9ae5-home\") pod \"distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg\" (UID: \"1e044c02-0053-40e8-9f80-e806cd2d9ae5\") " pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" Oct 30 21:42:22.027168 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:22.027122 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e044c02-0053-40e8-9f80-e806cd2d9ae5-tmp-dir\") pod \"distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg\" (UID: \"1e044c02-0053-40e8-9f80-e806cd2d9ae5\") " pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" Oct 30 21:42:22.027384 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:22.027225 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e044c02-0053-40e8-9f80-e806cd2d9ae5-model-cache\") pod \"distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg\" (UID: \"1e044c02-0053-40e8-9f80-e806cd2d9ae5\") " pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" Oct 30 21:42:22.027384 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:22.027260 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e044c02-0053-40e8-9f80-e806cd2d9ae5-dshm\") pod \"distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg\" (UID: \"1e044c02-0053-40e8-9f80-e806cd2d9ae5\") " pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" Oct 30 21:42:22.044237 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:22.044202 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ed76b8\"/\"distinct-w2-ed76b8-kserve-self-signed-certs\"" Oct 30 21:42:22.047710 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:22.047686 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg"] Oct 30 21:42:22.133361 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:22.130965 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e044c02-0053-40e8-9f80-e806cd2d9ae5-dshm\") pod \"distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg\" (UID: \"1e044c02-0053-40e8-9f80-e806cd2d9ae5\") " pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" Oct 30 21:42:22.133361 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:22.131064 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5t4c\" (UniqueName: \"kubernetes.io/projected/1e044c02-0053-40e8-9f80-e806cd2d9ae5-kube-api-access-k5t4c\") pod \"distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg\" (UID: \"1e044c02-0053-40e8-9f80-e806cd2d9ae5\") " pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" Oct 30 21:42:22.133361 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:22.131138 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e044c02-0053-40e8-9f80-e806cd2d9ae5-tls-certs\") pod \"distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg\" (UID: \"1e044c02-0053-40e8-9f80-e806cd2d9ae5\") " pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" Oct 30 21:42:22.133361 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:22.131183 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e044c02-0053-40e8-9f80-e806cd2d9ae5-home\") pod \"distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg\" (UID: \"1e044c02-0053-40e8-9f80-e806cd2d9ae5\") " pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" Oct 30 21:42:22.133361 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:22.131219 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e044c02-0053-40e8-9f80-e806cd2d9ae5-tmp-dir\") pod \"distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg\" (UID: \"1e044c02-0053-40e8-9f80-e806cd2d9ae5\") " pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" Oct 30 21:42:22.133361 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:22.131284 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e044c02-0053-40e8-9f80-e806cd2d9ae5-model-cache\") pod \"distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg\" (UID: \"1e044c02-0053-40e8-9f80-e806cd2d9ae5\") " pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" Oct 30 21:42:22.133361 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:22.131746 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e044c02-0053-40e8-9f80-e806cd2d9ae5-model-cache\") pod \"distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg\" (UID: \"1e044c02-0053-40e8-9f80-e806cd2d9ae5\") " pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" Oct 30 21:42:22.134651 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:22.134619 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e044c02-0053-40e8-9f80-e806cd2d9ae5-home\") pod \"distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg\" (UID: \"1e044c02-0053-40e8-9f80-e806cd2d9ae5\") " pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" Oct 30 21:42:22.135503 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:22.135448 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e044c02-0053-40e8-9f80-e806cd2d9ae5-tmp-dir\") pod \"distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg\" (UID: \"1e044c02-0053-40e8-9f80-e806cd2d9ae5\") " pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" Oct 30 21:42:22.139304 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:22.139283 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e044c02-0053-40e8-9f80-e806cd2d9ae5-dshm\") pod \"distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg\" (UID: \"1e044c02-0053-40e8-9f80-e806cd2d9ae5\") " pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" Oct 30 21:42:22.141403 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:22.141379 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e044c02-0053-40e8-9f80-e806cd2d9ae5-tls-certs\") pod \"distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg\" (UID: \"1e044c02-0053-40e8-9f80-e806cd2d9ae5\") " pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" Oct 30 21:42:22.193104 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:22.193030 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5t4c\" (UniqueName: \"kubernetes.io/projected/1e044c02-0053-40e8-9f80-e806cd2d9ae5-kube-api-access-k5t4c\") pod \"distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg\" (UID: \"1e044c02-0053-40e8-9f80-e806cd2d9ae5\") " pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" Oct 30 21:42:22.251396 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:22.251353 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" Oct 30 21:42:22.709885 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:22.709849 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg"] Oct 30 21:42:22.714123 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:42:22.714094 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e044c02_0053_40e8_9f80_e806cd2d9ae5.slice/crio-4be83a0f00718aab11f3451ce360c2ad8431f09e63b504a0943361e0cf39f50b WatchSource:0}: Error finding container 4be83a0f00718aab11f3451ce360c2ad8431f09e63b504a0943361e0cf39f50b: Status 404 returned error can't find the container with id 4be83a0f00718aab11f3451ce360c2ad8431f09e63b504a0943361e0cf39f50b Oct 30 21:42:23.443611 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:23.443580 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" event={"ID":"1e044c02-0053-40e8-9f80-e806cd2d9ae5","Type":"ContainerStarted","Data":"9dd25fa7195cbf6728318117ab0300f29ff576231a5b8c4958bf02c277efe16a"} Oct 30 21:42:23.444037 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:23.443625 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" event={"ID":"1e044c02-0053-40e8-9f80-e806cd2d9ae5","Type":"ContainerStarted","Data":"4be83a0f00718aab11f3451ce360c2ad8431f09e63b504a0943361e0cf39f50b"} Oct 30 21:42:23.555761 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:23.555704 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" podStartSLOduration=2.555688239 podStartE2EDuration="2.555688239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:23.554857995 +0000 UTC m=+1719.109557772" watchObservedRunningTime="2026-10-30 21:42:23.555688239 +0000 UTC m=+1719.110388017" Oct 30 21:42:24.251693 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:24.251647 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" Oct 30 21:42:32.251673 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:32.251632 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" Oct 30 21:42:32.264714 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:32.264681 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" Oct 30 21:42:32.510568 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:32.510497 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" Oct 30 21:42:37.466780 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:37.466748 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w"] Oct 30 21:42:37.470450 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:37.470424 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" Oct 30 21:42:37.566712 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:37.566681 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-afbe5b\"/\"distinct-2-w0-afbe5b-kserve-self-signed-certs\"" Oct 30 21:42:37.571089 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:37.571059 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w"] Oct 30 21:42:37.623361 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:37.616743 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e2d7535-075d-4211-b814-35d6abff7605-dshm\") pod \"distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w\" (UID: \"9e2d7535-075d-4211-b814-35d6abff7605\") " pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" Oct 30 21:42:37.623361 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:37.616795 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e2d7535-075d-4211-b814-35d6abff7605-tls-certs\") pod \"distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w\" (UID: \"9e2d7535-075d-4211-b814-35d6abff7605\") " pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" Oct 30 21:42:37.623361 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:37.616863 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e2d7535-075d-4211-b814-35d6abff7605-tmp-dir\") pod \"distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w\" (UID: \"9e2d7535-075d-4211-b814-35d6abff7605\") " pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" Oct 30 21:42:37.623361 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:37.616910 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e2d7535-075d-4211-b814-35d6abff7605-model-cache\") pod \"distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w\" (UID: \"9e2d7535-075d-4211-b814-35d6abff7605\") " pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" Oct 30 21:42:37.623361 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:37.616947 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e2d7535-075d-4211-b814-35d6abff7605-home\") pod \"distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w\" (UID: \"9e2d7535-075d-4211-b814-35d6abff7605\") " pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" Oct 30 21:42:37.623361 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:37.616985 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbnml\" (UniqueName: \"kubernetes.io/projected/9e2d7535-075d-4211-b814-35d6abff7605-kube-api-access-nbnml\") pod \"distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w\" (UID: \"9e2d7535-075d-4211-b814-35d6abff7605\") " pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" Oct 30 21:42:37.718398 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:37.718298 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e2d7535-075d-4211-b814-35d6abff7605-tmp-dir\") pod \"distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w\" (UID: \"9e2d7535-075d-4211-b814-35d6abff7605\") " pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" Oct 30 21:42:37.718398 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:37.718371 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e2d7535-075d-4211-b814-35d6abff7605-model-cache\") pod \"distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w\" (UID: \"9e2d7535-075d-4211-b814-35d6abff7605\") " pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" Oct 30 21:42:37.718637 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:37.718401 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e2d7535-075d-4211-b814-35d6abff7605-home\") pod \"distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w\" (UID: \"9e2d7535-075d-4211-b814-35d6abff7605\") " pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" Oct 30 21:42:37.718637 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:37.718610 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbnml\" (UniqueName: \"kubernetes.io/projected/9e2d7535-075d-4211-b814-35d6abff7605-kube-api-access-nbnml\") pod \"distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w\" (UID: \"9e2d7535-075d-4211-b814-35d6abff7605\") " pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" Oct 30 21:42:37.718894 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:37.718869 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e2d7535-075d-4211-b814-35d6abff7605-model-cache\") pod \"distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w\" (UID: \"9e2d7535-075d-4211-b814-35d6abff7605\") " pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" Oct 30 21:42:37.718980 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:37.718961 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e2d7535-075d-4211-b814-35d6abff7605-home\") pod \"distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w\" (UID: \"9e2d7535-075d-4211-b814-35d6abff7605\") " pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" Oct 30 21:42:37.719079 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:37.719062 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e2d7535-075d-4211-b814-35d6abff7605-dshm\") pod \"distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w\" (UID: \"9e2d7535-075d-4211-b814-35d6abff7605\") " pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" Oct 30 21:42:37.719141 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:37.719087 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e2d7535-075d-4211-b814-35d6abff7605-tls-certs\") pod \"distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w\" (UID: \"9e2d7535-075d-4211-b814-35d6abff7605\") " pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" Oct 30 21:42:37.719316 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:37.719291 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e2d7535-075d-4211-b814-35d6abff7605-tmp-dir\") pod \"distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w\" (UID: \"9e2d7535-075d-4211-b814-35d6abff7605\") " pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" Oct 30 21:42:37.721534 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:37.721509 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e2d7535-075d-4211-b814-35d6abff7605-tls-certs\") pod \"distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w\" (UID: \"9e2d7535-075d-4211-b814-35d6abff7605\") " pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" Oct 30 21:42:37.721636 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:37.721556 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e2d7535-075d-4211-b814-35d6abff7605-dshm\") pod \"distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w\" (UID: \"9e2d7535-075d-4211-b814-35d6abff7605\") " pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" Oct 30 21:42:37.776755 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:37.776722 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbnml\" (UniqueName: \"kubernetes.io/projected/9e2d7535-075d-4211-b814-35d6abff7605-kube-api-access-nbnml\") pod \"distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w\" (UID: \"9e2d7535-075d-4211-b814-35d6abff7605\") " pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" Oct 30 21:42:37.783421 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:37.783383 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" Oct 30 21:42:38.172426 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:38.172389 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w"] Oct 30 21:42:38.177487 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:42:38.177460 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e2d7535_075d_4211_b814_35d6abff7605.slice/crio-b2c14f67d09e7f4dd951ead410c40532d7e2c8d18112374d9e0fe059057a3cd0 WatchSource:0}: Error finding container b2c14f67d09e7f4dd951ead410c40532d7e2c8d18112374d9e0fe059057a3cd0: Status 404 returned error can't find the container with id b2c14f67d09e7f4dd951ead410c40532d7e2c8d18112374d9e0fe059057a3cd0 Oct 30 21:42:38.538202 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:38.538164 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" event={"ID":"9e2d7535-075d-4211-b814-35d6abff7605","Type":"ContainerStarted","Data":"464a950cb3eb7661e8c2a30c1b06fd05892a0b474bffc35fdfd16a2bf907abb6"} Oct 30 21:42:38.538202 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:38.538205 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" event={"ID":"9e2d7535-075d-4211-b814-35d6abff7605","Type":"ContainerStarted","Data":"b2c14f67d09e7f4dd951ead410c40532d7e2c8d18112374d9e0fe059057a3cd0"} Oct 30 21:42:38.652222 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:38.652170 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" podStartSLOduration=1.6521577189999999 podStartE2EDuration="1.652157719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:38.649532104 +0000 UTC m=+1734.204231884" watchObservedRunningTime="2026-10-30 21:42:38.652157719 +0000 UTC m=+1734.206857496" Oct 30 21:42:38.795415 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:38.788858 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" Oct 30 21:42:41.858201 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:41.858168 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:42:46.439035 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:46.439002 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6"] Oct 30 21:42:46.442961 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:46.442941 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" Oct 30 21:42:46.542647 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:46.542614 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6"] Oct 30 21:42:46.626808 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:46.626775 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0802666-5926-4a4f-a438-8e932494758c-credential-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6\" (UID: \"a0802666-5926-4a4f-a438-8e932494758c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" Oct 30 21:42:46.626991 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:46.626902 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0802666-5926-4a4f-a438-8e932494758c-istio-data\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6\" (UID: \"a0802666-5926-4a4f-a438-8e932494758c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" Oct 30 21:42:46.626991 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:46.626950 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0802666-5926-4a4f-a438-8e932494758c-workload-certs\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6\" (UID: \"a0802666-5926-4a4f-a438-8e932494758c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" Oct 30 21:42:46.626991 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:46.626967 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkfvn\" (UniqueName: \"kubernetes.io/projected/a0802666-5926-4a4f-a438-8e932494758c-kube-api-access-nkfvn\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6\" (UID: \"a0802666-5926-4a4f-a438-8e932494758c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" Oct 30 21:42:46.627093 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:46.626993 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0802666-5926-4a4f-a438-8e932494758c-istio-envoy\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6\" (UID: \"a0802666-5926-4a4f-a438-8e932494758c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" Oct 30 21:42:46.627093 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:46.627065 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0802666-5926-4a4f-a438-8e932494758c-istiod-ca-cert\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6\" (UID: \"a0802666-5926-4a4f-a438-8e932494758c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" Oct 30 21:42:46.627163 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:46.627138 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0802666-5926-4a4f-a438-8e932494758c-workload-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6\" (UID: \"a0802666-5926-4a4f-a438-8e932494758c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" Oct 30 21:42:46.627199 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:46.627180 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0802666-5926-4a4f-a438-8e932494758c-istio-podinfo\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6\" (UID: \"a0802666-5926-4a4f-a438-8e932494758c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" Oct 30 21:42:46.627298 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:46.627278 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0802666-5926-4a4f-a438-8e932494758c-istio-token\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6\" (UID: \"a0802666-5926-4a4f-a438-8e932494758c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" Oct 30 21:42:46.728556 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:46.728509 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0802666-5926-4a4f-a438-8e932494758c-istiod-ca-cert\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6\" (UID: \"a0802666-5926-4a4f-a438-8e932494758c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" Oct 30 21:42:46.728761 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:46.728575 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0802666-5926-4a4f-a438-8e932494758c-workload-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6\" (UID: \"a0802666-5926-4a4f-a438-8e932494758c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" Oct 30 21:42:46.728761 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:46.728632 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0802666-5926-4a4f-a438-8e932494758c-istio-podinfo\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6\" (UID: \"a0802666-5926-4a4f-a438-8e932494758c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" Oct 30 21:42:46.728761 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:46.728684 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0802666-5926-4a4f-a438-8e932494758c-istio-token\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6\" (UID: \"a0802666-5926-4a4f-a438-8e932494758c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" Oct 30 21:42:46.728761 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:46.728742 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0802666-5926-4a4f-a438-8e932494758c-credential-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6\" (UID: \"a0802666-5926-4a4f-a438-8e932494758c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" Oct 30 21:42:46.728971 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:46.728821 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0802666-5926-4a4f-a438-8e932494758c-istio-data\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6\" (UID: \"a0802666-5926-4a4f-a438-8e932494758c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" Oct 30 21:42:46.728971 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:46.728876 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0802666-5926-4a4f-a438-8e932494758c-workload-certs\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6\" (UID: \"a0802666-5926-4a4f-a438-8e932494758c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" Oct 30 21:42:46.728971 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:46.728902 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkfvn\" (UniqueName: \"kubernetes.io/projected/a0802666-5926-4a4f-a438-8e932494758c-kube-api-access-nkfvn\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6\" (UID: \"a0802666-5926-4a4f-a438-8e932494758c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" Oct 30 21:42:46.728971 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:46.728944 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0802666-5926-4a4f-a438-8e932494758c-istio-envoy\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6\" (UID: \"a0802666-5926-4a4f-a438-8e932494758c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" Oct 30 21:42:46.729172 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:46.729032 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0802666-5926-4a4f-a438-8e932494758c-workload-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6\" (UID: \"a0802666-5926-4a4f-a438-8e932494758c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" Oct 30 21:42:46.729232 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:46.729192 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0802666-5926-4a4f-a438-8e932494758c-istio-data\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6\" (UID: \"a0802666-5926-4a4f-a438-8e932494758c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" Oct 30 21:42:46.729289 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:46.729228 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0802666-5926-4a4f-a438-8e932494758c-credential-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6\" (UID: \"a0802666-5926-4a4f-a438-8e932494758c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" Oct 30 21:42:46.729396 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:46.729379 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0802666-5926-4a4f-a438-8e932494758c-workload-certs\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6\" (UID: \"a0802666-5926-4a4f-a438-8e932494758c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" Oct 30 21:42:46.729439 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:46.729408 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0802666-5926-4a4f-a438-8e932494758c-istiod-ca-cert\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6\" (UID: \"a0802666-5926-4a4f-a438-8e932494758c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" Oct 30 21:42:46.731196 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:46.731167 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0802666-5926-4a4f-a438-8e932494758c-istio-envoy\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6\" (UID: \"a0802666-5926-4a4f-a438-8e932494758c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" Oct 30 21:42:46.731307 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:46.731202 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0802666-5926-4a4f-a438-8e932494758c-istio-podinfo\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6\" (UID: \"a0802666-5926-4a4f-a438-8e932494758c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" Oct 30 21:42:46.786220 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:46.786186 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0802666-5926-4a4f-a438-8e932494758c-istio-token\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6\" (UID: \"a0802666-5926-4a4f-a438-8e932494758c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" Oct 30 21:42:46.786354 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:46.786315 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkfvn\" (UniqueName: \"kubernetes.io/projected/a0802666-5926-4a4f-a438-8e932494758c-kube-api-access-nkfvn\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6\" (UID: \"a0802666-5926-4a4f-a438-8e932494758c\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" Oct 30 21:42:47.056538 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.056462 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" Oct 30 21:42:47.349566 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.349468 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2"] Oct 30 21:42:47.354502 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.354478 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" Oct 30 21:42:47.404217 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.404182 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-71aab5-openshift-default-dockercfg-hxtfl\"" Oct 30 21:42:47.414679 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.414649 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2"] Oct 30 21:42:47.534957 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.534922 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6"] Oct 30 21:42:47.537289 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.537263 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c80c2c-2622-43a7-ab5c-bea1f6089c90-workload-socket\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2\" (UID: \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" Oct 30 21:42:47.537442 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.537422 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5nfc\" (UniqueName: \"kubernetes.io/projected/52c80c2c-2622-43a7-ab5c-bea1f6089c90-kube-api-access-p5nfc\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2\" (UID: \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" Oct 30 21:42:47.537537 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.537514 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c80c2c-2622-43a7-ab5c-bea1f6089c90-istio-data\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2\" (UID: \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" Oct 30 21:42:47.537592 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.537572 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c80c2c-2622-43a7-ab5c-bea1f6089c90-istio-envoy\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2\" (UID: \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" Oct 30 21:42:47.537592 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.537589 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52c80c2c-2622-43a7-ab5c-bea1f6089c90-istio-token\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2\" (UID: \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" Oct 30 21:42:47.537677 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.537608 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c80c2c-2622-43a7-ab5c-bea1f6089c90-istiod-ca-cert\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2\" (UID: \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" Oct 30 21:42:47.537746 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.537732 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c80c2c-2622-43a7-ab5c-bea1f6089c90-credential-socket\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2\" (UID: \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" Oct 30 21:42:47.537797 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.537768 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c80c2c-2622-43a7-ab5c-bea1f6089c90-workload-certs\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2\" (UID: \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" Oct 30 21:42:47.537885 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.537862 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c80c2c-2622-43a7-ab5c-bea1f6089c90-istio-podinfo\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2\" (UID: \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" Oct 30 21:42:47.538614 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:42:47.538589 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0802666_5926_4a4f_a438_8e932494758c.slice/crio-87c124acef9d44ad4b251c3c37006c3870b74a5158f352cb8adabe1cd8af32a9 WatchSource:0}: Error finding container 87c124acef9d44ad4b251c3c37006c3870b74a5158f352cb8adabe1cd8af32a9: Status 404 returned error can't find the container with id 87c124acef9d44ad4b251c3c37006c3870b74a5158f352cb8adabe1cd8af32a9 Oct 30 21:42:47.541164 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.541132 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:42:47.541260 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.541195 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:42:47.541260 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.541224 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:42:47.608878 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.608807 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" event={"ID":"a0802666-5926-4a4f-a438-8e932494758c","Type":"ContainerStarted","Data":"87c124acef9d44ad4b251c3c37006c3870b74a5158f352cb8adabe1cd8af32a9"} Oct 30 21:42:47.639460 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.639416 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5nfc\" (UniqueName: \"kubernetes.io/projected/52c80c2c-2622-43a7-ab5c-bea1f6089c90-kube-api-access-p5nfc\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2\" (UID: \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" Oct 30 21:42:47.639633 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.639493 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c80c2c-2622-43a7-ab5c-bea1f6089c90-istio-data\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2\" (UID: \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" Oct 30 21:42:47.639633 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.639524 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c80c2c-2622-43a7-ab5c-bea1f6089c90-istio-envoy\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2\" (UID: \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" Oct 30 21:42:47.639633 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.639548 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52c80c2c-2622-43a7-ab5c-bea1f6089c90-istio-token\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2\" (UID: \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" Oct 30 21:42:47.639633 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.639575 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c80c2c-2622-43a7-ab5c-bea1f6089c90-istiod-ca-cert\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2\" (UID: \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" Oct 30 21:42:47.639912 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.639653 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c80c2c-2622-43a7-ab5c-bea1f6089c90-credential-socket\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2\" (UID: \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" Oct 30 21:42:47.639912 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.639695 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c80c2c-2622-43a7-ab5c-bea1f6089c90-workload-certs\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2\" (UID: \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" Oct 30 21:42:47.639912 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.639737 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c80c2c-2622-43a7-ab5c-bea1f6089c90-istio-podinfo\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2\" (UID: \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" Oct 30 21:42:47.639912 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.639788 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c80c2c-2622-43a7-ab5c-bea1f6089c90-workload-socket\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2\" (UID: \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" Oct 30 21:42:47.639912 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.639905 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c80c2c-2622-43a7-ab5c-bea1f6089c90-istio-data\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2\" (UID: \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" Oct 30 21:42:47.640183 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.640119 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c80c2c-2622-43a7-ab5c-bea1f6089c90-workload-socket\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2\" (UID: \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" Oct 30 21:42:47.640183 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.640166 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c80c2c-2622-43a7-ab5c-bea1f6089c90-credential-socket\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2\" (UID: \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" Oct 30 21:42:47.640289 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.640216 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c80c2c-2622-43a7-ab5c-bea1f6089c90-workload-certs\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2\" (UID: \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" Oct 30 21:42:47.640722 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.640439 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c80c2c-2622-43a7-ab5c-bea1f6089c90-istiod-ca-cert\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2\" (UID: \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" Oct 30 21:42:47.642510 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.642487 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c80c2c-2622-43a7-ab5c-bea1f6089c90-istio-envoy\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2\" (UID: \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" Oct 30 21:42:47.642645 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.642616 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c80c2c-2622-43a7-ab5c-bea1f6089c90-istio-podinfo\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2\" (UID: \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" Oct 30 21:42:47.699830 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.699792 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52c80c2c-2622-43a7-ab5c-bea1f6089c90-istio-token\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2\" (UID: \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" Oct 30 21:42:47.700117 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.700092 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5nfc\" (UniqueName: \"kubernetes.io/projected/52c80c2c-2622-43a7-ab5c-bea1f6089c90-kube-api-access-p5nfc\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2\" (UID: \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" Oct 30 21:42:47.784078 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.784037 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" Oct 30 21:42:47.798058 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.798026 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" Oct 30 21:42:47.968576 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:47.968494 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" Oct 30 21:42:48.417767 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:48.417730 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2"] Oct 30 21:42:48.423548 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:42:48.423513 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c80c2c_2622_43a7_ab5c_bea1f6089c90.slice/crio-3c9cc18108960889030c048b02920a15e19d9c379d217376568128a583aa80e1 WatchSource:0}: Error finding container 3c9cc18108960889030c048b02920a15e19d9c379d217376568128a583aa80e1: Status 404 returned error can't find the container with id 3c9cc18108960889030c048b02920a15e19d9c379d217376568128a583aa80e1 Oct 30 21:42:48.426296 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:48.426254 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:42:48.426423 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:48.426356 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:42:48.426423 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:48.426413 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:42:48.619730 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:48.619692 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" event={"ID":"52c80c2c-2622-43a7-ab5c-bea1f6089c90","Type":"ContainerStarted","Data":"506ec9cf0df4614b55fb5fae142b98fbc40cc9034d23ce84e022554813446fbe"} Oct 30 21:42:48.620152 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:48.619755 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" event={"ID":"52c80c2c-2622-43a7-ab5c-bea1f6089c90","Type":"ContainerStarted","Data":"3c9cc18108960889030c048b02920a15e19d9c379d217376568128a583aa80e1"} Oct 30 21:42:48.621790 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:48.621763 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" event={"ID":"a0802666-5926-4a4f-a438-8e932494758c","Type":"ContainerStarted","Data":"9e12acd38ecb1e266db2ec5f3db45fce431c0094c6e7044baa990d1f514cec6f"} Oct 30 21:42:48.671474 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:48.669979 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" Oct 30 21:42:48.750367 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:48.746457 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" podStartSLOduration=1.746432697 podStartE2EDuration="1.746432697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:48.7408792 +0000 UTC m=+1744.295579013" watchObservedRunningTime="2026-10-30 21:42:48.746432697 +0000 UTC m=+1744.301132468" Oct 30 21:42:48.971985 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:48.971484 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" Oct 30 21:42:48.989439 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:48.989352 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" podStartSLOduration=2.989317229 podStartE2EDuration="2.989317229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:48.866795813 +0000 UTC m=+1744.421495591" watchObservedRunningTime="2026-10-30 21:42:48.989317229 +0000 UTC m=+1744.544017007" Oct 30 21:42:49.066795 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:49.066037 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" Oct 30 21:42:49.066795 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:49.066430 4560 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:42:49.066795 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:49.066485 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" podUID="a0802666-5926-4a4f-a438-8e932494758c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 21:42:49.708360 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:42:49.707475 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:42:49.974546 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:49.974448 4560 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:42:49.974806 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:49.974773 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" podUID="52c80c2c-2622-43a7-ab5c-bea1f6089c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 21:42:50.075365 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:50.064452 4560 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:42:50.075365 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:50.064515 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" podUID="a0802666-5926-4a4f-a438-8e932494758c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 21:42:50.977028 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:50.976441 4560 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:42:50.977028 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:50.976510 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" podUID="52c80c2c-2622-43a7-ab5c-bea1f6089c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 21:42:51.057911 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:51.057785 4560 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:42:51.057911 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:51.057850 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" podUID="a0802666-5926-4a4f-a438-8e932494758c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 21:42:51.970757 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:51.970713 4560 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:42:51.971089 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:51.970789 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" podUID="52c80c2c-2622-43a7-ab5c-bea1f6089c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 21:42:52.058364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:52.057796 4560 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:42:52.058364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:52.057871 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" podUID="a0802666-5926-4a4f-a438-8e932494758c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 21:42:52.973365 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:52.971698 4560 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:42:52.973365 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:52.971772 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" podUID="52c80c2c-2622-43a7-ab5c-bea1f6089c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 21:42:53.063364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:53.060597 4560 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:42:53.063364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:53.060656 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" podUID="a0802666-5926-4a4f-a438-8e932494758c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 21:42:53.969074 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:53.969032 4560 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:42:53.969258 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:53.969106 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" podUID="52c80c2c-2622-43a7-ab5c-bea1f6089c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 21:42:54.058024 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:54.057845 4560 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:42:54.058024 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:54.057916 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" podUID="a0802666-5926-4a4f-a438-8e932494758c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 21:42:54.969353 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:54.969311 4560 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:42:54.969756 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:54.969401 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" podUID="52c80c2c-2622-43a7-ab5c-bea1f6089c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 21:42:55.063243 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:55.063196 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" Oct 30 21:42:55.063868 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:55.063828 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" Oct 30 21:42:55.065120 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:55.065094 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" Oct 30 21:42:55.973280 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:55.973251 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" Oct 30 21:42:55.973770 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:55.973749 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" Oct 30 21:42:55.974509 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:42:55.974493 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" Oct 30 21:43:11.610307 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:11.610268 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:43:17.836865 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:17.836830 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0872-v8dcn"] Oct 30 21:43:17.840734 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:17.840714 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0872-v8dcn" Oct 30 21:43:17.898362 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:17.898311 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w5dt\" (UniqueName: \"kubernetes.io/projected/20465276-05de-4b00-80fa-d40c49a55d0e-kube-api-access-2w5dt\") pod \"maas-api-revoke-sub-7e6a0872-v8dcn\" (UID: \"20465276-05de-4b00-80fa-d40c49a55d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0872-v8dcn" Oct 30 21:43:17.898512 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:17.898406 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20465276-05de-4b00-80fa-d40c49a55d0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e6a0872-v8dcn\" (UID: \"20465276-05de-4b00-80fa-d40c49a55d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0872-v8dcn" Oct 30 21:43:17.898512 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:17.898445 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20465276-05de-4b00-80fa-d40c49a55d0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e6a0872-v8dcn\" (UID: \"20465276-05de-4b00-80fa-d40c49a55d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0872-v8dcn" Oct 30 21:43:17.938786 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:17.938759 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:43:17.938786 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:17.938766 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:43:17.939018 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:17.938792 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:43:17.939018 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:17.938852 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:43:17.942044 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:17.942024 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0872-v8dcn"] Oct 30 21:43:18.000187 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:18.000156 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20465276-05de-4b00-80fa-d40c49a55d0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e6a0872-v8dcn\" (UID: \"20465276-05de-4b00-80fa-d40c49a55d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0872-v8dcn" Oct 30 21:43:18.000301 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:18.000194 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20465276-05de-4b00-80fa-d40c49a55d0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e6a0872-v8dcn\" (UID: \"20465276-05de-4b00-80fa-d40c49a55d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0872-v8dcn" Oct 30 21:43:18.000381 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:18.000365 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w5dt\" (UniqueName: \"kubernetes.io/projected/20465276-05de-4b00-80fa-d40c49a55d0e-kube-api-access-2w5dt\") pod \"maas-api-revoke-sub-7e6a0872-v8dcn\" (UID: \"20465276-05de-4b00-80fa-d40c49a55d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0872-v8dcn" Oct 30 21:43:18.000986 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:18.000956 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20465276-05de-4b00-80fa-d40c49a55d0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e6a0872-v8dcn\" (UID: \"20465276-05de-4b00-80fa-d40c49a55d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0872-v8dcn" Oct 30 21:43:18.002520 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:18.002503 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20465276-05de-4b00-80fa-d40c49a55d0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e6a0872-v8dcn\" (UID: \"20465276-05de-4b00-80fa-d40c49a55d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0872-v8dcn" Oct 30 21:43:18.058120 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:18.058098 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w5dt\" (UniqueName: \"kubernetes.io/projected/20465276-05de-4b00-80fa-d40c49a55d0e-kube-api-access-2w5dt\") pod \"maas-api-revoke-sub-7e6a0872-v8dcn\" (UID: \"20465276-05de-4b00-80fa-d40c49a55d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0872-v8dcn" Oct 30 21:43:18.154456 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:18.154388 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0872-v8dcn" Oct 30 21:43:18.527674 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:18.527626 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0872-v8dcn"] Oct 30 21:43:18.532550 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:43:18.532527 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20465276_05de_4b00_80fa_d40c49a55d0e.slice/crio-66a5d0ca18450cc56309b5f9e56be7286dd28f49ab305cc9210c3dec98c53528 WatchSource:0}: Error finding container 66a5d0ca18450cc56309b5f9e56be7286dd28f49ab305cc9210c3dec98c53528: Status 404 returned error can't find the container with id 66a5d0ca18450cc56309b5f9e56be7286dd28f49ab305cc9210c3dec98c53528 Oct 30 21:43:18.873398 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:18.873371 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0872-v8dcn" event={"ID":"20465276-05de-4b00-80fa-d40c49a55d0e","Type":"ContainerStarted","Data":"95a81b44ce200d767e61087e8b1cc2d6fda96886b8d55efb4389ee975acc3e42"} Oct 30 21:43:18.873720 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:18.873420 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0872-v8dcn" event={"ID":"20465276-05de-4b00-80fa-d40c49a55d0e","Type":"ContainerStarted","Data":"66a5d0ca18450cc56309b5f9e56be7286dd28f49ab305cc9210c3dec98c53528"} Oct 30 21:43:18.983049 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:18.982993 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0872-v8dcn" podStartSLOduration=1.982980134 podStartE2EDuration="1.982980134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:18.982268333 +0000 UTC m=+1774.536968136" watchObservedRunningTime="2026-10-30 21:43:18.982980134 +0000 UTC m=+1774.537679910" Oct 30 21:43:19.686298 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:43:19.686213 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:43:19.881475 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:19.881438 4560 generic.go:356] "Generic (PLEG): container finished" podID="20465276-05de-4b00-80fa-d40c49a55d0e" containerID="95a81b44ce200d767e61087e8b1cc2d6fda96886b8d55efb4389ee975acc3e42" exitCode=0 Oct 30 21:43:19.881835 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:19.881495 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0872-v8dcn" event={"ID":"20465276-05de-4b00-80fa-d40c49a55d0e","Type":"ContainerDied","Data":"95a81b44ce200d767e61087e8b1cc2d6fda96886b8d55efb4389ee975acc3e42"} Oct 30 21:43:21.030017 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:21.029994 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0872-v8dcn" Oct 30 21:43:21.135456 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:21.135428 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20465276-05de-4b00-80fa-d40c49a55d0e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20465276-05de-4b00-80fa-d40c49a55d0e-maas-api-openshift-ca\") pod \"20465276-05de-4b00-80fa-d40c49a55d0e\" (UID: \"20465276-05de-4b00-80fa-d40c49a55d0e\") " Oct 30 21:43:21.135598 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:21.135493 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20465276-05de-4b00-80fa-d40c49a55d0e-kube-api-access-2w5dt\" (UniqueName: \"kubernetes.io/projected/20465276-05de-4b00-80fa-d40c49a55d0e-kube-api-access-2w5dt\") pod \"20465276-05de-4b00-80fa-d40c49a55d0e\" (UID: \"20465276-05de-4b00-80fa-d40c49a55d0e\") " Oct 30 21:43:21.135598 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:21.135543 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20465276-05de-4b00-80fa-d40c49a55d0e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20465276-05de-4b00-80fa-d40c49a55d0e-maas-api-xks-ca\") pod \"20465276-05de-4b00-80fa-d40c49a55d0e\" (UID: \"20465276-05de-4b00-80fa-d40c49a55d0e\") " Oct 30 21:43:21.135767 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:21.135745 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20465276-05de-4b00-80fa-d40c49a55d0e-maas-api-openshift-ca" pod "20465276-05de-4b00-80fa-d40c49a55d0e" (UID: "20465276-05de-4b00-80fa-d40c49a55d0e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:43:21.135886 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:21.135873 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20465276-05de-4b00-80fa-d40c49a55d0e-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:21.137497 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:21.137477 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20465276-05de-4b00-80fa-d40c49a55d0e-maas-api-xks-ca" pod "20465276-05de-4b00-80fa-d40c49a55d0e" (UID: "20465276-05de-4b00-80fa-d40c49a55d0e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:43:21.137637 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:21.137618 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20465276-05de-4b00-80fa-d40c49a55d0e-kube-api-access-2w5dt" pod "20465276-05de-4b00-80fa-d40c49a55d0e" (UID: "20465276-05de-4b00-80fa-d40c49a55d0e"). InnerVolumeSpecName "kube-api-access-2w5dt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:43:21.256829 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:21.256781 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20465276-05de-4b00-80fa-d40c49a55d0e-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:21.257078 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:21.257057 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w5dt\" (UniqueName: \"kubernetes.io/projected/20465276-05de-4b00-80fa-d40c49a55d0e-kube-api-access-2w5dt\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:21.894789 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:21.894764 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0872-v8dcn" Oct 30 21:43:21.894975 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:21.894758 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0872-v8dcn" event={"ID":"20465276-05de-4b00-80fa-d40c49a55d0e","Type":"ContainerDied","Data":"66a5d0ca18450cc56309b5f9e56be7286dd28f49ab305cc9210c3dec98c53528"} Oct 30 21:43:21.894975 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:21.894890 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66a5d0ca18450cc56309b5f9e56be7286dd28f49ab305cc9210c3dec98c53528" Oct 30 21:43:48.058354 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:48.058314 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7378b486-6p5r9"] Oct 30 21:43:48.105256 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:48.105230 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7378b486-6p5r9"] Oct 30 21:43:48.575657 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:48.575622 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc5a7f12-8474-49fa-b5ba-b3db497c0483" path="/var/lib/kubelet/pods/dc5a7f12-8474-49fa-b5ba-b3db497c0483/volumes" Oct 30 21:43:49.191839 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:49.191804 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41ff2a70-2frst"] Oct 30 21:43:49.192266 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:49.192247 4560 cpu_manager.go:414] "containerMap: removing container" podUID="20465276-05de-4b00-80fa-d40c49a55d0e" containerName="revoke-keys" Oct 30 21:43:49.192266 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:49.192261 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20465276-05de-4b00-80fa-d40c49a55d0e" containerName="revoke-keys" Oct 30 21:43:49.192403 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:49.192392 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="20465276-05de-4b00-80fa-d40c49a55d0e" containerName="revoke-keys" Oct 30 21:43:49.196016 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:49.195989 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ff2a70-2frst" Oct 30 21:43:49.268123 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:49.268089 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9361faed-bd30-4786-acb4-0dafcd9a87db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41ff2a70-2frst\" (UID: \"9361faed-bd30-4786-acb4-0dafcd9a87db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ff2a70-2frst" Oct 30 21:43:49.268277 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:49.268149 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9361faed-bd30-4786-acb4-0dafcd9a87db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41ff2a70-2frst\" (UID: \"9361faed-bd30-4786-acb4-0dafcd9a87db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ff2a70-2frst" Oct 30 21:43:49.268368 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:49.268327 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwjch\" (UniqueName: \"kubernetes.io/projected/9361faed-bd30-4786-acb4-0dafcd9a87db-kube-api-access-kwjch\") pod \"maas-api-revoke-sub-41ff2a70-2frst\" (UID: \"9361faed-bd30-4786-acb4-0dafcd9a87db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ff2a70-2frst" Oct 30 21:43:49.292695 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:49.292668 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:43:49.292695 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:49.292692 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:43:49.292867 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:49.292678 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:43:49.292867 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:49.292762 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:43:49.296219 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:49.296198 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41ff2a70-2frst"] Oct 30 21:43:49.369611 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:49.369589 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwjch\" (UniqueName: \"kubernetes.io/projected/9361faed-bd30-4786-acb4-0dafcd9a87db-kube-api-access-kwjch\") pod \"maas-api-revoke-sub-41ff2a70-2frst\" (UID: \"9361faed-bd30-4786-acb4-0dafcd9a87db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ff2a70-2frst" Oct 30 21:43:49.369723 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:49.369671 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9361faed-bd30-4786-acb4-0dafcd9a87db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41ff2a70-2frst\" (UID: \"9361faed-bd30-4786-acb4-0dafcd9a87db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ff2a70-2frst" Oct 30 21:43:49.369723 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:49.369711 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9361faed-bd30-4786-acb4-0dafcd9a87db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41ff2a70-2frst\" (UID: \"9361faed-bd30-4786-acb4-0dafcd9a87db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ff2a70-2frst" Oct 30 21:43:49.370286 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:49.370264 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9361faed-bd30-4786-acb4-0dafcd9a87db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41ff2a70-2frst\" (UID: \"9361faed-bd30-4786-acb4-0dafcd9a87db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ff2a70-2frst" Oct 30 21:43:49.372072 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:49.372049 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9361faed-bd30-4786-acb4-0dafcd9a87db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41ff2a70-2frst\" (UID: \"9361faed-bd30-4786-acb4-0dafcd9a87db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ff2a70-2frst" Oct 30 21:43:49.427068 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:49.427044 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwjch\" (UniqueName: \"kubernetes.io/projected/9361faed-bd30-4786-acb4-0dafcd9a87db-kube-api-access-kwjch\") pod \"maas-api-revoke-sub-41ff2a70-2frst\" (UID: \"9361faed-bd30-4786-acb4-0dafcd9a87db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ff2a70-2frst" Oct 30 21:43:49.508876 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:49.508855 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ff2a70-2frst" Oct 30 21:43:49.686418 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:43:49.686308 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:43:49.884538 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:49.884413 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41ff2a70-2frst"] Oct 30 21:43:49.888954 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:43:49.888930 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9361faed_bd30_4786_acb4_0dafcd9a87db.slice/crio-04d48cec7b2e6973284952fbe2e635894167835e881845ff20dff1bc8e0c5f81 WatchSource:0}: Error finding container 04d48cec7b2e6973284952fbe2e635894167835e881845ff20dff1bc8e0c5f81: Status 404 returned error can't find the container with id 04d48cec7b2e6973284952fbe2e635894167835e881845ff20dff1bc8e0c5f81 Oct 30 21:43:50.061072 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:50.061034 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ff2a70-2frst" event={"ID":"9361faed-bd30-4786-acb4-0dafcd9a87db","Type":"ContainerStarted","Data":"04d48cec7b2e6973284952fbe2e635894167835e881845ff20dff1bc8e0c5f81"} Oct 30 21:43:50.185699 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:50.185665 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:43:51.068876 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:51.068794 4560 generic.go:356] "Generic (PLEG): container finished" podID="9361faed-bd30-4786-acb4-0dafcd9a87db" containerID="344ff5e284e30ec0d9d96e9248ec3b175b529167a9650694f03b863e161ebe87" exitCode=0 Oct 30 21:43:51.069249 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:51.068882 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ff2a70-2frst" event={"ID":"9361faed-bd30-4786-acb4-0dafcd9a87db","Type":"ContainerDied","Data":"344ff5e284e30ec0d9d96e9248ec3b175b529167a9650694f03b863e161ebe87"} Oct 30 21:43:52.218442 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:52.218419 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ff2a70-2frst" Oct 30 21:43:52.300175 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:52.300149 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9361faed-bd30-4786-acb4-0dafcd9a87db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9361faed-bd30-4786-acb4-0dafcd9a87db-maas-api-openshift-ca\") pod \"9361faed-bd30-4786-acb4-0dafcd9a87db\" (UID: \"9361faed-bd30-4786-acb4-0dafcd9a87db\") " Oct 30 21:43:52.300325 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:52.300195 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9361faed-bd30-4786-acb4-0dafcd9a87db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9361faed-bd30-4786-acb4-0dafcd9a87db-maas-api-xks-ca\") pod \"9361faed-bd30-4786-acb4-0dafcd9a87db\" (UID: \"9361faed-bd30-4786-acb4-0dafcd9a87db\") " Oct 30 21:43:52.300325 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:52.300238 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9361faed-bd30-4786-acb4-0dafcd9a87db-kube-api-access-kwjch\" (UniqueName: \"kubernetes.io/projected/9361faed-bd30-4786-acb4-0dafcd9a87db-kube-api-access-kwjch\") pod \"9361faed-bd30-4786-acb4-0dafcd9a87db\" (UID: \"9361faed-bd30-4786-acb4-0dafcd9a87db\") " Oct 30 21:43:52.300591 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:52.300567 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9361faed-bd30-4786-acb4-0dafcd9a87db-maas-api-openshift-ca" pod "9361faed-bd30-4786-acb4-0dafcd9a87db" (UID: "9361faed-bd30-4786-acb4-0dafcd9a87db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:43:52.300697 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:52.300677 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9361faed-bd30-4786-acb4-0dafcd9a87db-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:52.302326 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:52.302304 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9361faed-bd30-4786-acb4-0dafcd9a87db-kube-api-access-kwjch" pod "9361faed-bd30-4786-acb4-0dafcd9a87db" (UID: "9361faed-bd30-4786-acb4-0dafcd9a87db"). InnerVolumeSpecName "kube-api-access-kwjch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:43:52.302445 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:52.302425 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9361faed-bd30-4786-acb4-0dafcd9a87db-maas-api-xks-ca" pod "9361faed-bd30-4786-acb4-0dafcd9a87db" (UID: "9361faed-bd30-4786-acb4-0dafcd9a87db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:43:52.402277 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:52.402195 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwjch\" (UniqueName: \"kubernetes.io/projected/9361faed-bd30-4786-acb4-0dafcd9a87db-kube-api-access-kwjch\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:52.402277 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:52.402229 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9361faed-bd30-4786-acb4-0dafcd9a87db-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:53.081239 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:53.081206 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ff2a70-2frst" Oct 30 21:43:53.081239 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:53.081230 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ff2a70-2frst" event={"ID":"9361faed-bd30-4786-acb4-0dafcd9a87db","Type":"ContainerDied","Data":"04d48cec7b2e6973284952fbe2e635894167835e881845ff20dff1bc8e0c5f81"} Oct 30 21:43:53.081465 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:43:53.081278 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04d48cec7b2e6973284952fbe2e635894167835e881845ff20dff1bc8e0c5f81" Oct 30 21:44:08.694562 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:08.694529 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8aa35089-qv44j"] Oct 30 21:44:08.695061 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:08.694957 4560 cpu_manager.go:414] "containerMap: removing container" podUID="9361faed-bd30-4786-acb4-0dafcd9a87db" containerName="revoke-keys" Oct 30 21:44:08.695061 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:08.694969 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9361faed-bd30-4786-acb4-0dafcd9a87db" containerName="revoke-keys" Oct 30 21:44:08.695061 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:08.695059 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="9361faed-bd30-4786-acb4-0dafcd9a87db" containerName="revoke-keys" Oct 30 21:44:08.698407 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:08.698384 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa35089-qv44j" Oct 30 21:44:08.708687 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:08.708663 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/181d6638-2a2e-4718-bd93-a8df3cdb528b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8aa35089-qv44j\" (UID: \"181d6638-2a2e-4718-bd93-a8df3cdb528b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa35089-qv44j" Oct 30 21:44:08.708874 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:08.708856 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8b4x\" (UniqueName: \"kubernetes.io/projected/181d6638-2a2e-4718-bd93-a8df3cdb528b-kube-api-access-q8b4x\") pod \"maas-api-revoke-sub-8aa35089-qv44j\" (UID: \"181d6638-2a2e-4718-bd93-a8df3cdb528b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa35089-qv44j" Oct 30 21:44:08.708915 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:08.708895 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/181d6638-2a2e-4718-bd93-a8df3cdb528b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8aa35089-qv44j\" (UID: \"181d6638-2a2e-4718-bd93-a8df3cdb528b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa35089-qv44j" Oct 30 21:44:08.794213 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:08.794174 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:44:08.794394 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:08.794214 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:44:08.794394 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:08.794251 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:44:08.794394 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:08.794293 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:44:08.797728 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:08.797708 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8aa35089-qv44j"] Oct 30 21:44:08.810208 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:08.810179 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8b4x\" (UniqueName: \"kubernetes.io/projected/181d6638-2a2e-4718-bd93-a8df3cdb528b-kube-api-access-q8b4x\") pod \"maas-api-revoke-sub-8aa35089-qv44j\" (UID: \"181d6638-2a2e-4718-bd93-a8df3cdb528b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa35089-qv44j" Oct 30 21:44:08.810333 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:08.810292 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/181d6638-2a2e-4718-bd93-a8df3cdb528b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8aa35089-qv44j\" (UID: \"181d6638-2a2e-4718-bd93-a8df3cdb528b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa35089-qv44j" Oct 30 21:44:08.810410 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:08.810375 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/181d6638-2a2e-4718-bd93-a8df3cdb528b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8aa35089-qv44j\" (UID: \"181d6638-2a2e-4718-bd93-a8df3cdb528b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa35089-qv44j" Oct 30 21:44:08.811050 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:08.811032 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/181d6638-2a2e-4718-bd93-a8df3cdb528b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8aa35089-qv44j\" (UID: \"181d6638-2a2e-4718-bd93-a8df3cdb528b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa35089-qv44j" Oct 30 21:44:08.812697 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:08.812680 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/181d6638-2a2e-4718-bd93-a8df3cdb528b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8aa35089-qv44j\" (UID: \"181d6638-2a2e-4718-bd93-a8df3cdb528b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa35089-qv44j" Oct 30 21:44:08.867837 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:08.867808 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8b4x\" (UniqueName: \"kubernetes.io/projected/181d6638-2a2e-4718-bd93-a8df3cdb528b-kube-api-access-q8b4x\") pod \"maas-api-revoke-sub-8aa35089-qv44j\" (UID: \"181d6638-2a2e-4718-bd93-a8df3cdb528b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa35089-qv44j" Oct 30 21:44:09.012435 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:09.012401 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa35089-qv44j" Oct 30 21:44:09.463697 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:09.463664 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8aa35089-qv44j"] Oct 30 21:44:09.465933 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:44:09.465897 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod181d6638_2a2e_4718_bd93_a8df3cdb528b.slice/crio-c2aeccc4426ddfb4a7176257cb267e1fb5bd46bebf94b100cf920cf44830c5df WatchSource:0}: Error finding container c2aeccc4426ddfb4a7176257cb267e1fb5bd46bebf94b100cf920cf44830c5df: Status 404 returned error can't find the container with id c2aeccc4426ddfb4a7176257cb267e1fb5bd46bebf94b100cf920cf44830c5df Oct 30 21:44:10.202459 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:10.202357 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa35089-qv44j" event={"ID":"181d6638-2a2e-4718-bd93-a8df3cdb528b","Type":"ContainerStarted","Data":"6b2f8fad10de1dea3c3a91319938717e31fcda5e3ec094adcff1be1efe18d7c4"} Oct 30 21:44:10.202459 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:10.202431 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa35089-qv44j" event={"ID":"181d6638-2a2e-4718-bd93-a8df3cdb528b","Type":"ContainerStarted","Data":"c2aeccc4426ddfb4a7176257cb267e1fb5bd46bebf94b100cf920cf44830c5df"} Oct 30 21:44:11.210074 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:11.210041 4560 generic.go:356] "Generic (PLEG): container finished" podID="181d6638-2a2e-4718-bd93-a8df3cdb528b" containerID="6b2f8fad10de1dea3c3a91319938717e31fcda5e3ec094adcff1be1efe18d7c4" exitCode=0 Oct 30 21:44:11.210530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:11.210133 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa35089-qv44j" event={"ID":"181d6638-2a2e-4718-bd93-a8df3cdb528b","Type":"ContainerDied","Data":"6b2f8fad10de1dea3c3a91319938717e31fcda5e3ec094adcff1be1efe18d7c4"} Oct 30 21:44:12.365283 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:12.365261 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa35089-qv44j" Oct 30 21:44:12.464053 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:12.464025 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/181d6638-2a2e-4718-bd93-a8df3cdb528b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/181d6638-2a2e-4718-bd93-a8df3cdb528b-maas-api-xks-ca\") pod \"181d6638-2a2e-4718-bd93-a8df3cdb528b\" (UID: \"181d6638-2a2e-4718-bd93-a8df3cdb528b\") " Oct 30 21:44:12.464192 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:12.464136 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/181d6638-2a2e-4718-bd93-a8df3cdb528b-kube-api-access-q8b4x\" (UniqueName: \"kubernetes.io/projected/181d6638-2a2e-4718-bd93-a8df3cdb528b-kube-api-access-q8b4x\") pod \"181d6638-2a2e-4718-bd93-a8df3cdb528b\" (UID: \"181d6638-2a2e-4718-bd93-a8df3cdb528b\") " Oct 30 21:44:12.464192 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:12.464165 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/181d6638-2a2e-4718-bd93-a8df3cdb528b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/181d6638-2a2e-4718-bd93-a8df3cdb528b-maas-api-openshift-ca\") pod \"181d6638-2a2e-4718-bd93-a8df3cdb528b\" (UID: \"181d6638-2a2e-4718-bd93-a8df3cdb528b\") " Oct 30 21:44:12.464561 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:12.464534 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/181d6638-2a2e-4718-bd93-a8df3cdb528b-maas-api-openshift-ca" pod "181d6638-2a2e-4718-bd93-a8df3cdb528b" (UID: "181d6638-2a2e-4718-bd93-a8df3cdb528b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:12.466117 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:12.466084 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/181d6638-2a2e-4718-bd93-a8df3cdb528b-kube-api-access-q8b4x" pod "181d6638-2a2e-4718-bd93-a8df3cdb528b" (UID: "181d6638-2a2e-4718-bd93-a8df3cdb528b"). InnerVolumeSpecName "kube-api-access-q8b4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:12.466230 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:12.466186 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/181d6638-2a2e-4718-bd93-a8df3cdb528b-maas-api-xks-ca" pod "181d6638-2a2e-4718-bd93-a8df3cdb528b" (UID: "181d6638-2a2e-4718-bd93-a8df3cdb528b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:12.565977 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:12.565946 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8b4x\" (UniqueName: \"kubernetes.io/projected/181d6638-2a2e-4718-bd93-a8df3cdb528b-kube-api-access-q8b4x\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:12.565977 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:12.565970 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/181d6638-2a2e-4718-bd93-a8df3cdb528b-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:12.565977 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:12.565981 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/181d6638-2a2e-4718-bd93-a8df3cdb528b-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:12.688352 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:12.688317 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9"] Oct 30 21:44:12.688836 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:12.688808 4560 cpu_manager.go:414] "containerMap: removing container" podUID="181d6638-2a2e-4718-bd93-a8df3cdb528b" containerName="revoke-keys" Oct 30 21:44:12.688836 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:12.688827 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="181d6638-2a2e-4718-bd93-a8df3cdb528b" containerName="revoke-keys" Oct 30 21:44:12.688960 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:12.688938 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="181d6638-2a2e-4718-bd93-a8df3cdb528b" containerName="revoke-keys" Oct 30 21:44:12.692195 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:12.692180 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" Oct 30 21:44:12.746800 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:12.746770 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a2dced\"/\"distinct-2-w6-a2dced-kserve-self-signed-certs\"" Oct 30 21:44:12.750138 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:12.750113 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9"] Oct 30 21:44:12.768290 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:12.768267 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b80b0fba-3830-4a74-a724-aead35443b89-home\") pod \"distinct-2-w6-a2dced-kserve-897cb76c-lz5w9\" (UID: \"b80b0fba-3830-4a74-a724-aead35443b89\") " pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" Oct 30 21:44:12.768408 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:12.768387 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b80b0fba-3830-4a74-a724-aead35443b89-model-cache\") pod \"distinct-2-w6-a2dced-kserve-897cb76c-lz5w9\" (UID: \"b80b0fba-3830-4a74-a724-aead35443b89\") " pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" Oct 30 21:44:12.768479 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:12.768416 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b80b0fba-3830-4a74-a724-aead35443b89-tmp-dir\") pod \"distinct-2-w6-a2dced-kserve-897cb76c-lz5w9\" (UID: \"b80b0fba-3830-4a74-a724-aead35443b89\") " pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" Oct 30 21:44:12.768557 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:12.768537 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b80b0fba-3830-4a74-a724-aead35443b89-dshm\") pod \"distinct-2-w6-a2dced-kserve-897cb76c-lz5w9\" (UID: \"b80b0fba-3830-4a74-a724-aead35443b89\") " pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" Oct 30 21:44:12.768644 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:12.768627 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gnlf\" (UniqueName: \"kubernetes.io/projected/b80b0fba-3830-4a74-a724-aead35443b89-kube-api-access-6gnlf\") pod \"distinct-2-w6-a2dced-kserve-897cb76c-lz5w9\" (UID: \"b80b0fba-3830-4a74-a724-aead35443b89\") " pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" Oct 30 21:44:12.768700 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:12.768688 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b80b0fba-3830-4a74-a724-aead35443b89-tls-certs\") pod \"distinct-2-w6-a2dced-kserve-897cb76c-lz5w9\" (UID: \"b80b0fba-3830-4a74-a724-aead35443b89\") " pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" Oct 30 21:44:12.870394 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:12.870351 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b80b0fba-3830-4a74-a724-aead35443b89-model-cache\") pod \"distinct-2-w6-a2dced-kserve-897cb76c-lz5w9\" (UID: \"b80b0fba-3830-4a74-a724-aead35443b89\") " pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" Oct 30 21:44:12.871165 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:12.871144 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b80b0fba-3830-4a74-a724-aead35443b89-tmp-dir\") pod \"distinct-2-w6-a2dced-kserve-897cb76c-lz5w9\" (UID: \"b80b0fba-3830-4a74-a724-aead35443b89\") " pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" Oct 30 21:44:12.871779 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:12.871759 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b80b0fba-3830-4a74-a724-aead35443b89-dshm\") pod \"distinct-2-w6-a2dced-kserve-897cb76c-lz5w9\" (UID: \"b80b0fba-3830-4a74-a724-aead35443b89\") " pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" Oct 30 21:44:12.872453 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:12.872430 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gnlf\" (UniqueName: \"kubernetes.io/projected/b80b0fba-3830-4a74-a724-aead35443b89-kube-api-access-6gnlf\") pod \"distinct-2-w6-a2dced-kserve-897cb76c-lz5w9\" (UID: \"b80b0fba-3830-4a74-a724-aead35443b89\") " pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" Oct 30 21:44:12.873036 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:12.873015 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b80b0fba-3830-4a74-a724-aead35443b89-tls-certs\") pod \"distinct-2-w6-a2dced-kserve-897cb76c-lz5w9\" (UID: \"b80b0fba-3830-4a74-a724-aead35443b89\") " pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" Oct 30 21:44:12.873640 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:12.873622 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b80b0fba-3830-4a74-a724-aead35443b89-home\") pod \"distinct-2-w6-a2dced-kserve-897cb76c-lz5w9\" (UID: \"b80b0fba-3830-4a74-a724-aead35443b89\") " pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" Oct 30 21:44:12.874226 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:12.871612 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b80b0fba-3830-4a74-a724-aead35443b89-tmp-dir\") pod \"distinct-2-w6-a2dced-kserve-897cb76c-lz5w9\" (UID: \"b80b0fba-3830-4a74-a724-aead35443b89\") " pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" Oct 30 21:44:12.874381 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:12.874048 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b80b0fba-3830-4a74-a724-aead35443b89-home\") pod \"distinct-2-w6-a2dced-kserve-897cb76c-lz5w9\" (UID: \"b80b0fba-3830-4a74-a724-aead35443b89\") " pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" Oct 30 21:44:12.874495 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:12.871079 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b80b0fba-3830-4a74-a724-aead35443b89-model-cache\") pod \"distinct-2-w6-a2dced-kserve-897cb76c-lz5w9\" (UID: \"b80b0fba-3830-4a74-a724-aead35443b89\") " pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" Oct 30 21:44:12.886693 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:12.886664 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b80b0fba-3830-4a74-a724-aead35443b89-dshm\") pod \"distinct-2-w6-a2dced-kserve-897cb76c-lz5w9\" (UID: \"b80b0fba-3830-4a74-a724-aead35443b89\") " pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" Oct 30 21:44:12.887885 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:12.887857 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b80b0fba-3830-4a74-a724-aead35443b89-tls-certs\") pod \"distinct-2-w6-a2dced-kserve-897cb76c-lz5w9\" (UID: \"b80b0fba-3830-4a74-a724-aead35443b89\") " pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" Oct 30 21:44:12.930404 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:12.930370 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gnlf\" (UniqueName: \"kubernetes.io/projected/b80b0fba-3830-4a74-a724-aead35443b89-kube-api-access-6gnlf\") pod \"distinct-2-w6-a2dced-kserve-897cb76c-lz5w9\" (UID: \"b80b0fba-3830-4a74-a724-aead35443b89\") " pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" Oct 30 21:44:13.003789 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:13.003696 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" Oct 30 21:44:13.223975 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:13.223948 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa35089-qv44j" Oct 30 21:44:13.224170 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:13.223982 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa35089-qv44j" event={"ID":"181d6638-2a2e-4718-bd93-a8df3cdb528b","Type":"ContainerDied","Data":"c2aeccc4426ddfb4a7176257cb267e1fb5bd46bebf94b100cf920cf44830c5df"} Oct 30 21:44:13.224170 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:13.224047 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2aeccc4426ddfb4a7176257cb267e1fb5bd46bebf94b100cf920cf44830c5df" Oct 30 21:44:13.405663 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:13.405590 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9"] Oct 30 21:44:13.410585 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:44:13.410559 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb80b0fba_3830_4a74_a724_aead35443b89.slice/crio-2b72ee34d5d7cdcb1ff87a8af0e694744e65051a3c066d4463d6e0a527310b8b WatchSource:0}: Error finding container 2b72ee34d5d7cdcb1ff87a8af0e694744e65051a3c066d4463d6e0a527310b8b: Status 404 returned error can't find the container with id 2b72ee34d5d7cdcb1ff87a8af0e694744e65051a3c066d4463d6e0a527310b8b Oct 30 21:44:14.058025 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:14.057990 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35f9b9ef-8xlqp"] Oct 30 21:44:14.104235 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:14.104206 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35f9b9ef-8xlqp"] Oct 30 21:44:14.240223 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:14.240183 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" event={"ID":"b80b0fba-3830-4a74-a724-aead35443b89","Type":"ContainerStarted","Data":"77ebdf1dfd8681c99ec738e440f0b16ceca7abea33d89c730c9cfdedfc551370"} Oct 30 21:44:14.240428 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:14.240230 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" event={"ID":"b80b0fba-3830-4a74-a724-aead35443b89","Type":"ContainerStarted","Data":"2b72ee34d5d7cdcb1ff87a8af0e694744e65051a3c066d4463d6e0a527310b8b"} Oct 30 21:44:14.354333 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:14.354231 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" podStartSLOduration=2.354214304 podStartE2EDuration="2.354214304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:14.350801516 +0000 UTC m=+1829.905501294" watchObservedRunningTime="2026-10-30 21:44:14.354214304 +0000 UTC m=+1829.908914081" Oct 30 21:44:14.575852 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:14.575819 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f104d1a-2722-4dc1-be18-12a7786b22c9" path="/var/lib/kubelet/pods/3f104d1a-2722-4dc1-be18-12a7786b22c9/volumes" Oct 30 21:44:15.004686 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:15.004656 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" Oct 30 21:44:17.066368 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:17.065551 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddda40cb-sb7df"] Oct 30 21:44:17.100666 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:17.100640 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddda40cb-sb7df"] Oct 30 21:44:18.578391 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:18.577601 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="678fb080-b0c0-4967-9943-409845f0b31c" path="/var/lib/kubelet/pods/678fb080-b0c0-4967-9943-409845f0b31c/volumes" Oct 30 21:44:19.684371 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:44:19.684311 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:44:23.004507 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:23.004473 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" Oct 30 21:44:23.017365 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:23.017330 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" Oct 30 21:44:23.303989 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:23.303899 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" Oct 30 21:44:29.100511 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:29.100436 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6861dafe-b5bfq"] Oct 30 21:44:29.147520 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:29.147487 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6861dafe-b5bfq"] Oct 30 21:44:30.576178 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:30.576146 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc375f8c-fc71-46ed-84a1-58413a18eaac" path="/var/lib/kubelet/pods/cc375f8c-fc71-46ed-84a1-58413a18eaac/volumes" Oct 30 21:44:32.909647 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:32.909566 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn"] Oct 30 21:44:32.915300 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:32.915278 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" Oct 30 21:44:32.936616 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:32.936589 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-istio-data\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn\" (UID: \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" Oct 30 21:44:32.936716 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:32.936620 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-credential-socket\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn\" (UID: \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" Oct 30 21:44:32.936761 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:32.936735 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-workload-certs\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn\" (UID: \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" Oct 30 21:44:32.936804 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:32.936777 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-istio-envoy\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn\" (UID: \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" Oct 30 21:44:32.936844 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:32.936828 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-workload-socket\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn\" (UID: \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" Oct 30 21:44:32.936884 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:32.936855 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gjn7\" (UniqueName: \"kubernetes.io/projected/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-kube-api-access-2gjn7\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn\" (UID: \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" Oct 30 21:44:32.936932 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:32.936889 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-istio-podinfo\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn\" (UID: \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" Oct 30 21:44:32.936973 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:32.936947 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-istio-token\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn\" (UID: \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" Oct 30 21:44:32.937014 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:32.936982 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-istiod-ca-cert\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn\" (UID: \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" Oct 30 21:44:33.014965 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.014934 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn"] Oct 30 21:44:33.037846 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.037815 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-istio-podinfo\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn\" (UID: \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" Oct 30 21:44:33.037846 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.037849 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-istio-token\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn\" (UID: \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" Oct 30 21:44:33.038035 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.037868 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-istiod-ca-cert\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn\" (UID: \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" Oct 30 21:44:33.038035 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.037938 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-istio-data\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn\" (UID: \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" Oct 30 21:44:33.038035 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.037963 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-credential-socket\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn\" (UID: \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" Oct 30 21:44:33.038192 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.038061 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-workload-certs\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn\" (UID: \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" Oct 30 21:44:33.038192 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.038109 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-istio-envoy\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn\" (UID: \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" Oct 30 21:44:33.038192 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.038154 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-workload-socket\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn\" (UID: \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" Oct 30 21:44:33.038379 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.038199 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gjn7\" (UniqueName: \"kubernetes.io/projected/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-kube-api-access-2gjn7\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn\" (UID: \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" Oct 30 21:44:33.038646 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.038596 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-credential-socket\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn\" (UID: \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" Oct 30 21:44:33.038766 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.038666 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-workload-certs\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn\" (UID: \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" Oct 30 21:44:33.038766 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.038733 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-istiod-ca-cert\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn\" (UID: \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" Oct 30 21:44:33.038949 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.038925 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-workload-socket\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn\" (UID: \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" Oct 30 21:44:33.039038 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.038969 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-istio-data\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn\" (UID: \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" Oct 30 21:44:33.040530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.040506 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-istio-podinfo\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn\" (UID: \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" Oct 30 21:44:33.040631 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.040611 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-istio-envoy\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn\" (UID: \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" Oct 30 21:44:33.122647 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.122620 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-istio-token\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn\" (UID: \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" Oct 30 21:44:33.123919 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.123894 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gjn7\" (UniqueName: \"kubernetes.io/projected/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-kube-api-access-2gjn7\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn\" (UID: \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" Oct 30 21:44:33.228760 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.228730 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" Oct 30 21:44:33.516544 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.516469 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc"] Oct 30 21:44:33.521530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.521508 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" Oct 30 21:44:33.543501 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.543461 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1423e26a-6610-4adc-8a01-ebd58407a713-credential-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc\" (UID: \"1423e26a-6610-4adc-8a01-ebd58407a713\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" Oct 30 21:44:33.543634 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.543517 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1423e26a-6610-4adc-8a01-ebd58407a713-istio-data\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc\" (UID: \"1423e26a-6610-4adc-8a01-ebd58407a713\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" Oct 30 21:44:33.543634 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.543595 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1423e26a-6610-4adc-8a01-ebd58407a713-istiod-ca-cert\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc\" (UID: \"1423e26a-6610-4adc-8a01-ebd58407a713\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" Oct 30 21:44:33.543761 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.543653 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1423e26a-6610-4adc-8a01-ebd58407a713-workload-certs\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc\" (UID: \"1423e26a-6610-4adc-8a01-ebd58407a713\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" Oct 30 21:44:33.543761 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.543736 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnpgg\" (UniqueName: \"kubernetes.io/projected/1423e26a-6610-4adc-8a01-ebd58407a713-kube-api-access-gnpgg\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc\" (UID: \"1423e26a-6610-4adc-8a01-ebd58407a713\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" Oct 30 21:44:33.543851 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.543769 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1423e26a-6610-4adc-8a01-ebd58407a713-istio-token\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc\" (UID: \"1423e26a-6610-4adc-8a01-ebd58407a713\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" Oct 30 21:44:33.543885 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.543848 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1423e26a-6610-4adc-8a01-ebd58407a713-istio-podinfo\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc\" (UID: \"1423e26a-6610-4adc-8a01-ebd58407a713\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" Oct 30 21:44:33.543922 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.543904 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1423e26a-6610-4adc-8a01-ebd58407a713-istio-envoy\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc\" (UID: \"1423e26a-6610-4adc-8a01-ebd58407a713\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" Oct 30 21:44:33.543983 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.543971 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1423e26a-6610-4adc-8a01-ebd58407a713-workload-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc\" (UID: \"1423e26a-6610-4adc-8a01-ebd58407a713\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" Oct 30 21:44:33.565292 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.565264 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j"] Oct 30 21:44:33.569524 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.569502 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" Oct 30 21:44:33.602535 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.602503 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc"] Oct 30 21:44:33.641772 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.641738 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j"] Oct 30 21:44:33.645115 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.645088 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1423e26a-6610-4adc-8a01-ebd58407a713-workload-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc\" (UID: \"1423e26a-6610-4adc-8a01-ebd58407a713\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" Oct 30 21:44:33.645247 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.645145 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408e0e40-4d43-4ef8-adb0-07f371a17dd8-istiod-ca-cert\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j\" (UID: \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" Oct 30 21:44:33.645247 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.645193 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1423e26a-6610-4adc-8a01-ebd58407a713-credential-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc\" (UID: \"1423e26a-6610-4adc-8a01-ebd58407a713\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" Oct 30 21:44:33.645247 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.645213 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408e0e40-4d43-4ef8-adb0-07f371a17dd8-istio-data\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j\" (UID: \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" Oct 30 21:44:33.645445 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.645258 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1423e26a-6610-4adc-8a01-ebd58407a713-istio-data\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc\" (UID: \"1423e26a-6610-4adc-8a01-ebd58407a713\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" Oct 30 21:44:33.645445 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.645383 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1423e26a-6610-4adc-8a01-ebd58407a713-istiod-ca-cert\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc\" (UID: \"1423e26a-6610-4adc-8a01-ebd58407a713\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" Oct 30 21:44:33.645561 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.645517 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1423e26a-6610-4adc-8a01-ebd58407a713-workload-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc\" (UID: \"1423e26a-6610-4adc-8a01-ebd58407a713\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" Oct 30 21:44:33.645561 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.645547 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408e0e40-4d43-4ef8-adb0-07f371a17dd8-credential-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j\" (UID: \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" Oct 30 21:44:33.645665 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.645576 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1423e26a-6610-4adc-8a01-ebd58407a713-istio-data\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc\" (UID: \"1423e26a-6610-4adc-8a01-ebd58407a713\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" Oct 30 21:44:33.645665 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.645579 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvfb5\" (UniqueName: \"kubernetes.io/projected/408e0e40-4d43-4ef8-adb0-07f371a17dd8-kube-api-access-dvfb5\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j\" (UID: \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" Oct 30 21:44:33.645780 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.645669 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1423e26a-6610-4adc-8a01-ebd58407a713-workload-certs\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc\" (UID: \"1423e26a-6610-4adc-8a01-ebd58407a713\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" Oct 30 21:44:33.645780 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.645749 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnpgg\" (UniqueName: \"kubernetes.io/projected/1423e26a-6610-4adc-8a01-ebd58407a713-kube-api-access-gnpgg\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc\" (UID: \"1423e26a-6610-4adc-8a01-ebd58407a713\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" Oct 30 21:44:33.645895 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.645779 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1423e26a-6610-4adc-8a01-ebd58407a713-istio-token\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc\" (UID: \"1423e26a-6610-4adc-8a01-ebd58407a713\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" Oct 30 21:44:33.645895 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.645804 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408e0e40-4d43-4ef8-adb0-07f371a17dd8-istio-podinfo\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j\" (UID: \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" Oct 30 21:44:33.645895 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.645872 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1423e26a-6610-4adc-8a01-ebd58407a713-istio-podinfo\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc\" (UID: \"1423e26a-6610-4adc-8a01-ebd58407a713\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" Oct 30 21:44:33.645895 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.645751 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1423e26a-6610-4adc-8a01-ebd58407a713-credential-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc\" (UID: \"1423e26a-6610-4adc-8a01-ebd58407a713\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" Oct 30 21:44:33.646101 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.645908 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408e0e40-4d43-4ef8-adb0-07f371a17dd8-workload-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j\" (UID: \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" Oct 30 21:44:33.646101 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.645939 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408e0e40-4d43-4ef8-adb0-07f371a17dd8-workload-certs\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j\" (UID: \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" Oct 30 21:44:33.646101 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.645979 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408e0e40-4d43-4ef8-adb0-07f371a17dd8-istio-envoy\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j\" (UID: \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" Oct 30 21:44:33.646101 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.645996 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1423e26a-6610-4adc-8a01-ebd58407a713-workload-certs\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc\" (UID: \"1423e26a-6610-4adc-8a01-ebd58407a713\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" Oct 30 21:44:33.646101 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.646021 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1423e26a-6610-4adc-8a01-ebd58407a713-istio-envoy\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc\" (UID: \"1423e26a-6610-4adc-8a01-ebd58407a713\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" Oct 30 21:44:33.646101 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.646047 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408e0e40-4d43-4ef8-adb0-07f371a17dd8-istio-token\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j\" (UID: \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" Oct 30 21:44:33.646456 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.646238 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1423e26a-6610-4adc-8a01-ebd58407a713-istiod-ca-cert\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc\" (UID: \"1423e26a-6610-4adc-8a01-ebd58407a713\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" Oct 30 21:44:33.648547 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.648524 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1423e26a-6610-4adc-8a01-ebd58407a713-istio-envoy\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc\" (UID: \"1423e26a-6610-4adc-8a01-ebd58407a713\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" Oct 30 21:44:33.648785 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.648766 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1423e26a-6610-4adc-8a01-ebd58407a713-istio-podinfo\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc\" (UID: \"1423e26a-6610-4adc-8a01-ebd58407a713\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" Oct 30 21:44:33.700268 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.700235 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn"] Oct 30 21:44:33.708287 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:44:33.708260 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd64f50e0_cf7a_43ec_83ac_a73c0972cb39.slice/crio-1e14a284625adb5c924d96b024e79e13add413be7bd75a4f221f694b925f98dd WatchSource:0}: Error finding container 1e14a284625adb5c924d96b024e79e13add413be7bd75a4f221f694b925f98dd: Status 404 returned error can't find the container with id 1e14a284625adb5c924d96b024e79e13add413be7bd75a4f221f694b925f98dd Oct 30 21:44:33.708489 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.708467 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnpgg\" (UniqueName: \"kubernetes.io/projected/1423e26a-6610-4adc-8a01-ebd58407a713-kube-api-access-gnpgg\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc\" (UID: \"1423e26a-6610-4adc-8a01-ebd58407a713\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" Oct 30 21:44:33.710942 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.710906 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:44:33.711019 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.710983 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:44:33.711057 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.711023 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:44:33.717452 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.717429 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1423e26a-6610-4adc-8a01-ebd58407a713-istio-token\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc\" (UID: \"1423e26a-6610-4adc-8a01-ebd58407a713\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" Oct 30 21:44:33.746790 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.746763 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408e0e40-4d43-4ef8-adb0-07f371a17dd8-istio-data\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j\" (UID: \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" Oct 30 21:44:33.746893 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.746840 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408e0e40-4d43-4ef8-adb0-07f371a17dd8-credential-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j\" (UID: \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" Oct 30 21:44:33.746893 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.746861 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvfb5\" (UniqueName: \"kubernetes.io/projected/408e0e40-4d43-4ef8-adb0-07f371a17dd8-kube-api-access-dvfb5\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j\" (UID: \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" Oct 30 21:44:33.747004 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.746902 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408e0e40-4d43-4ef8-adb0-07f371a17dd8-istio-podinfo\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j\" (UID: \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" Oct 30 21:44:33.747004 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.746946 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408e0e40-4d43-4ef8-adb0-07f371a17dd8-workload-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j\" (UID: \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" Oct 30 21:44:33.747004 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.746963 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408e0e40-4d43-4ef8-adb0-07f371a17dd8-workload-certs\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j\" (UID: \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" Oct 30 21:44:33.747004 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.746991 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408e0e40-4d43-4ef8-adb0-07f371a17dd8-istio-envoy\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j\" (UID: \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" Oct 30 21:44:33.747220 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.747034 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408e0e40-4d43-4ef8-adb0-07f371a17dd8-istio-token\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j\" (UID: \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" Oct 30 21:44:33.747220 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.747129 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408e0e40-4d43-4ef8-adb0-07f371a17dd8-istiod-ca-cert\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j\" (UID: \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" Oct 30 21:44:33.747220 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.747155 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408e0e40-4d43-4ef8-adb0-07f371a17dd8-istio-data\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j\" (UID: \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" Oct 30 21:44:33.747382 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.747249 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408e0e40-4d43-4ef8-adb0-07f371a17dd8-credential-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j\" (UID: \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" Oct 30 21:44:33.747577 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.747553 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408e0e40-4d43-4ef8-adb0-07f371a17dd8-workload-socket\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j\" (UID: \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" Oct 30 21:44:33.747577 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.747568 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408e0e40-4d43-4ef8-adb0-07f371a17dd8-workload-certs\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j\" (UID: \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" Oct 30 21:44:33.747854 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.747827 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408e0e40-4d43-4ef8-adb0-07f371a17dd8-istiod-ca-cert\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j\" (UID: \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" Oct 30 21:44:33.749879 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.749853 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408e0e40-4d43-4ef8-adb0-07f371a17dd8-istio-podinfo\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j\" (UID: \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" Oct 30 21:44:33.750717 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.750696 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408e0e40-4d43-4ef8-adb0-07f371a17dd8-istio-envoy\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j\" (UID: \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" Oct 30 21:44:33.809879 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.809842 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvfb5\" (UniqueName: \"kubernetes.io/projected/408e0e40-4d43-4ef8-adb0-07f371a17dd8-kube-api-access-dvfb5\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j\" (UID: \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" Oct 30 21:44:33.815691 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.815665 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408e0e40-4d43-4ef8-adb0-07f371a17dd8-istio-token\") pod \"e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j\" (UID: \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\") " pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" Oct 30 21:44:33.835898 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.835874 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" Oct 30 21:44:33.883454 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:33.883409 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" Oct 30 21:44:34.046109 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:34.046026 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09bd6c1b-cp7jq"] Oct 30 21:44:34.051204 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:34.051183 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09bd6c1b-cp7jq" Oct 30 21:44:34.102386 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:34.102315 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:44:34.102530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:34.102455 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:44:34.104184 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:34.104158 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:44:34.104448 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:34.104427 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:44:34.113992 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:34.113966 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09bd6c1b-cp7jq"] Oct 30 21:44:34.152737 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:34.152703 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/633cda80-9a6a-4366-8dbd-8aa125655e2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09bd6c1b-cp7jq\" (UID: \"633cda80-9a6a-4366-8dbd-8aa125655e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09bd6c1b-cp7jq" Oct 30 21:44:34.153048 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:34.152783 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcwjs\" (UniqueName: \"kubernetes.io/projected/633cda80-9a6a-4366-8dbd-8aa125655e2d-kube-api-access-tcwjs\") pod \"maas-api-revoke-sub-09bd6c1b-cp7jq\" (UID: \"633cda80-9a6a-4366-8dbd-8aa125655e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09bd6c1b-cp7jq" Oct 30 21:44:34.153048 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:34.152898 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/633cda80-9a6a-4366-8dbd-8aa125655e2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09bd6c1b-cp7jq\" (UID: \"633cda80-9a6a-4366-8dbd-8aa125655e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09bd6c1b-cp7jq" Oct 30 21:44:34.226400 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:34.226366 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc"] Oct 30 21:44:34.233925 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:44:34.233897 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1423e26a_6610_4adc_8a01_ebd58407a713.slice/crio-646161b1a1ee147dd7d7d8a5389dc034c717a1e81d19b2876ea4eb14e747cf9f WatchSource:0}: Error finding container 646161b1a1ee147dd7d7d8a5389dc034c717a1e81d19b2876ea4eb14e747cf9f: Status 404 returned error can't find the container with id 646161b1a1ee147dd7d7d8a5389dc034c717a1e81d19b2876ea4eb14e747cf9f Oct 30 21:44:34.236429 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:34.236395 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:44:34.236514 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:34.236460 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:44:34.236514 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:34.236494 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:44:34.254254 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:34.254223 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/633cda80-9a6a-4366-8dbd-8aa125655e2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09bd6c1b-cp7jq\" (UID: \"633cda80-9a6a-4366-8dbd-8aa125655e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09bd6c1b-cp7jq" Oct 30 21:44:34.254419 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:34.254402 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/633cda80-9a6a-4366-8dbd-8aa125655e2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09bd6c1b-cp7jq\" (UID: \"633cda80-9a6a-4366-8dbd-8aa125655e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09bd6c1b-cp7jq" Oct 30 21:44:34.254503 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:34.254488 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcwjs\" (UniqueName: \"kubernetes.io/projected/633cda80-9a6a-4366-8dbd-8aa125655e2d-kube-api-access-tcwjs\") pod \"maas-api-revoke-sub-09bd6c1b-cp7jq\" (UID: \"633cda80-9a6a-4366-8dbd-8aa125655e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09bd6c1b-cp7jq" Oct 30 21:44:34.254969 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:34.254949 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/633cda80-9a6a-4366-8dbd-8aa125655e2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09bd6c1b-cp7jq\" (UID: \"633cda80-9a6a-4366-8dbd-8aa125655e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09bd6c1b-cp7jq" Oct 30 21:44:34.256719 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:34.256689 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/633cda80-9a6a-4366-8dbd-8aa125655e2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09bd6c1b-cp7jq\" (UID: \"633cda80-9a6a-4366-8dbd-8aa125655e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09bd6c1b-cp7jq" Oct 30 21:44:34.300101 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:34.300038 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j"] Oct 30 21:44:34.302211 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:44:34.302185 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod408e0e40_4d43_4ef8_adb0_07f371a17dd8.slice/crio-9046fbab9ad3e46df2a80271619df423704f0c5d194f5bcbfc310e349c14be32 WatchSource:0}: Error finding container 9046fbab9ad3e46df2a80271619df423704f0c5d194f5bcbfc310e349c14be32: Status 404 returned error can't find the container with id 9046fbab9ad3e46df2a80271619df423704f0c5d194f5bcbfc310e349c14be32 Oct 30 21:44:34.305253 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:34.305216 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:44:34.305390 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:34.305300 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:44:34.305390 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:34.305356 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:44:34.318911 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:34.318885 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcwjs\" (UniqueName: \"kubernetes.io/projected/633cda80-9a6a-4366-8dbd-8aa125655e2d-kube-api-access-tcwjs\") pod \"maas-api-revoke-sub-09bd6c1b-cp7jq\" (UID: \"633cda80-9a6a-4366-8dbd-8aa125655e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09bd6c1b-cp7jq" Oct 30 21:44:34.364501 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:34.364457 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" event={"ID":"d64f50e0-cf7a-43ec-83ac-a73c0972cb39","Type":"ContainerStarted","Data":"53e8673f6d93be057b26cf7786f8e4a7dde2460b4e1389dd43746eb2799934d4"} Oct 30 21:44:34.364662 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:34.364511 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" event={"ID":"d64f50e0-cf7a-43ec-83ac-a73c0972cb39","Type":"ContainerStarted","Data":"1e14a284625adb5c924d96b024e79e13add413be7bd75a4f221f694b925f98dd"} Oct 30 21:44:34.366406 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:34.366374 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" event={"ID":"1423e26a-6610-4adc-8a01-ebd58407a713","Type":"ContainerStarted","Data":"4524eaa22bbcb51637ed535f429ebf5cd042898e305a6184e06271ccaf7228a6"} Oct 30 21:44:34.366527 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:34.366430 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" event={"ID":"1423e26a-6610-4adc-8a01-ebd58407a713","Type":"ContainerStarted","Data":"646161b1a1ee147dd7d7d8a5389dc034c717a1e81d19b2876ea4eb14e747cf9f"} Oct 30 21:44:34.367930 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:34.367911 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" event={"ID":"408e0e40-4d43-4ef8-adb0-07f371a17dd8","Type":"ContainerStarted","Data":"9046fbab9ad3e46df2a80271619df423704f0c5d194f5bcbfc310e349c14be32"} Oct 30 21:44:34.374651 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:34.374630 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09bd6c1b-cp7jq" Oct 30 21:44:34.491964 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:34.491905 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" podStartSLOduration=2.491886106 podStartE2EDuration="2.491886106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:34.485918335 +0000 UTC m=+1850.040618141" watchObservedRunningTime="2026-10-30 21:44:34.491886106 +0000 UTC m=+1850.046585884" Oct 30 21:44:34.607841 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:34.607745 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" podStartSLOduration=1.607731308 podStartE2EDuration="1.607731308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:34.606807292 +0000 UTC m=+1850.161507071" watchObservedRunningTime="2026-10-30 21:44:34.607731308 +0000 UTC m=+1850.162431085" Oct 30 21:44:34.781207 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:34.781174 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09bd6c1b-cp7jq"] Oct 30 21:44:34.786912 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:44:34.786886 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod633cda80_9a6a_4366_8dbd_8aa125655e2d.slice/crio-ee2e5a67c9c2fda8fbc0b9e66289630dbe61bbbd948be87a070e4bf69ec51776 WatchSource:0}: Error finding container ee2e5a67c9c2fda8fbc0b9e66289630dbe61bbbd948be87a070e4bf69ec51776: Status 404 returned error can't find the container with id ee2e5a67c9c2fda8fbc0b9e66289630dbe61bbbd948be87a070e4bf69ec51776 Oct 30 21:44:34.844365 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:34.843328 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" Oct 30 21:44:35.238365 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:35.234400 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" Oct 30 21:44:35.239116 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:35.238834 4560 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 21:44:35.239116 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:35.238891 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" podUID="d64f50e0-cf7a-43ec-83ac-a73c0972cb39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 21:44:35.397365 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:35.396752 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09bd6c1b-cp7jq" event={"ID":"633cda80-9a6a-4366-8dbd-8aa125655e2d","Type":"ContainerStarted","Data":"5416453a2ce236a819cf445b54b509acf10bd74da949ddbd57ddc9a9d07b7bed"} Oct 30 21:44:35.397365 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:35.396819 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09bd6c1b-cp7jq" event={"ID":"633cda80-9a6a-4366-8dbd-8aa125655e2d","Type":"ContainerStarted","Data":"ee2e5a67c9c2fda8fbc0b9e66289630dbe61bbbd948be87a070e4bf69ec51776"} Oct 30 21:44:35.403593 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:35.403560 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" event={"ID":"408e0e40-4d43-4ef8-adb0-07f371a17dd8","Type":"ContainerStarted","Data":"e5fe40808d042be41381cb486e4fea21666d6ebdd287e51b10a4db08c289efb7"} Oct 30 21:44:35.517206 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:35.517106 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09bd6c1b-cp7jq" podStartSLOduration=1.517093557 podStartE2EDuration="1.517093557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:35.515129962 +0000 UTC m=+1851.069829737" watchObservedRunningTime="2026-10-30 21:44:35.517093557 +0000 UTC m=+1851.071793333" Oct 30 21:44:35.693363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:35.692765 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" podStartSLOduration=2.692737657 podStartE2EDuration="2.692737657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:35.636548592 +0000 UTC m=+1851.191248371" watchObservedRunningTime="2026-10-30 21:44:35.692737657 +0000 UTC m=+1851.247437435" Oct 30 21:44:35.858417 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:35.850795 4560 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 21:44:35.858417 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:35.850880 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" podUID="1423e26a-6610-4adc-8a01-ebd58407a713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 21:44:35.899776 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:35.896288 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" Oct 30 21:44:35.899776 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:35.896565 4560 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:44:35.899776 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:35.896614 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" podUID="408e0e40-4d43-4ef8-adb0-07f371a17dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 21:44:36.242364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:36.238756 4560 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 21:44:36.242364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:36.238820 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" podUID="d64f50e0-cf7a-43ec-83ac-a73c0972cb39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 21:44:36.411893 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:36.411864 4560 generic.go:356] "Generic (PLEG): container finished" podID="633cda80-9a6a-4366-8dbd-8aa125655e2d" containerID="5416453a2ce236a819cf445b54b509acf10bd74da949ddbd57ddc9a9d07b7bed" exitCode=0 Oct 30 21:44:36.412181 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:36.411934 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09bd6c1b-cp7jq" event={"ID":"633cda80-9a6a-4366-8dbd-8aa125655e2d","Type":"ContainerDied","Data":"5416453a2ce236a819cf445b54b509acf10bd74da949ddbd57ddc9a9d07b7bed"} Oct 30 21:44:36.847365 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:36.838788 4560 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 21:44:36.847365 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:36.838878 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" podUID="1423e26a-6610-4adc-8a01-ebd58407a713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 21:44:36.887364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:36.886785 4560 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:44:36.887364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:36.886853 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" podUID="408e0e40-4d43-4ef8-adb0-07f371a17dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 21:44:37.108412 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:37.092393 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf7cbcc7-8526x"] Oct 30 21:44:37.119390 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:37.115439 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf7cbcc7-8526x"] Oct 30 21:44:37.232364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:37.231581 4560 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 21:44:37.232364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:37.231653 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" podUID="d64f50e0-cf7a-43ec-83ac-a73c0972cb39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 21:44:37.845275 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:37.844815 4560 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 21:44:37.845275 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:37.844881 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" podUID="1423e26a-6610-4adc-8a01-ebd58407a713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 21:44:37.892081 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:37.891867 4560 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:44:37.892081 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:37.891928 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" podUID="408e0e40-4d43-4ef8-adb0-07f371a17dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 21:44:37.992438 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:37.977827 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09bd6c1b-cp7jq" Oct 30 21:44:38.029362 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:38.028597 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/633cda80-9a6a-4366-8dbd-8aa125655e2d-kube-api-access-tcwjs\" (UniqueName: \"kubernetes.io/projected/633cda80-9a6a-4366-8dbd-8aa125655e2d-kube-api-access-tcwjs\") pod \"633cda80-9a6a-4366-8dbd-8aa125655e2d\" (UID: \"633cda80-9a6a-4366-8dbd-8aa125655e2d\") " Oct 30 21:44:38.029362 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:38.028753 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/633cda80-9a6a-4366-8dbd-8aa125655e2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/633cda80-9a6a-4366-8dbd-8aa125655e2d-maas-api-xks-ca\") pod \"633cda80-9a6a-4366-8dbd-8aa125655e2d\" (UID: \"633cda80-9a6a-4366-8dbd-8aa125655e2d\") " Oct 30 21:44:38.029362 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:38.028900 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/633cda80-9a6a-4366-8dbd-8aa125655e2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/633cda80-9a6a-4366-8dbd-8aa125655e2d-maas-api-openshift-ca\") pod \"633cda80-9a6a-4366-8dbd-8aa125655e2d\" (UID: \"633cda80-9a6a-4366-8dbd-8aa125655e2d\") " Oct 30 21:44:38.043368 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:38.041558 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/633cda80-9a6a-4366-8dbd-8aa125655e2d-maas-api-openshift-ca" pod "633cda80-9a6a-4366-8dbd-8aa125655e2d" (UID: "633cda80-9a6a-4366-8dbd-8aa125655e2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:38.069681 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:38.069647 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/633cda80-9a6a-4366-8dbd-8aa125655e2d-maas-api-xks-ca" pod "633cda80-9a6a-4366-8dbd-8aa125655e2d" (UID: "633cda80-9a6a-4366-8dbd-8aa125655e2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:38.079726 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:38.078050 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/633cda80-9a6a-4366-8dbd-8aa125655e2d-kube-api-access-tcwjs" pod "633cda80-9a6a-4366-8dbd-8aa125655e2d" (UID: "633cda80-9a6a-4366-8dbd-8aa125655e2d"). InnerVolumeSpecName "kube-api-access-tcwjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:38.146472 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:38.130749 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcwjs\" (UniqueName: \"kubernetes.io/projected/633cda80-9a6a-4366-8dbd-8aa125655e2d-kube-api-access-tcwjs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:38.146472 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:38.130787 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/633cda80-9a6a-4366-8dbd-8aa125655e2d-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:38.146472 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:38.130803 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/633cda80-9a6a-4366-8dbd-8aa125655e2d-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:38.235364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:38.230797 4560 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 21:44:38.235364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:38.230876 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" podUID="d64f50e0-cf7a-43ec-83ac-a73c0972cb39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 21:44:38.460437 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:38.456937 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09bd6c1b-cp7jq" event={"ID":"633cda80-9a6a-4366-8dbd-8aa125655e2d","Type":"ContainerDied","Data":"ee2e5a67c9c2fda8fbc0b9e66289630dbe61bbbd948be87a070e4bf69ec51776"} Oct 30 21:44:38.460437 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:38.457035 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee2e5a67c9c2fda8fbc0b9e66289630dbe61bbbd948be87a070e4bf69ec51776" Oct 30 21:44:38.460437 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:38.457123 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09bd6c1b-cp7jq" Oct 30 21:44:38.720405 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:38.716568 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30" path="/var/lib/kubelet/pods/cba9ba1d-2d8d-4f89-a7e0-a8febf52cb30/volumes" Oct 30 21:44:38.840787 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:38.840747 4560 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 21:44:38.841123 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:38.841087 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" podUID="1423e26a-6610-4adc-8a01-ebd58407a713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 21:44:38.884746 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:38.884709 4560 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:44:38.885297 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:38.885265 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" podUID="408e0e40-4d43-4ef8-adb0-07f371a17dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 21:44:39.230738 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:39.230695 4560 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 21:44:39.231097 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:39.231064 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" podUID="d64f50e0-cf7a-43ec-83ac-a73c0972cb39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 21:44:39.847364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:39.846667 4560 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 21:44:39.847364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:39.846734 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" podUID="1423e26a-6610-4adc-8a01-ebd58407a713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 21:44:39.884795 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:39.884754 4560 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:44:39.885418 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:39.885385 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" podUID="408e0e40-4d43-4ef8-adb0-07f371a17dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 21:44:40.234867 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:40.234450 4560 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 21:44:40.234867 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:40.234519 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" podUID="d64f50e0-cf7a-43ec-83ac-a73c0972cb39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 21:44:40.847361 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:40.842697 4560 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 21:44:40.847361 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:40.842765 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" podUID="1423e26a-6610-4adc-8a01-ebd58407a713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 21:44:40.887722 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:40.887684 4560 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:44:40.888281 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:40.888246 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" podUID="408e0e40-4d43-4ef8-adb0-07f371a17dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 21:44:41.229315 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:41.229233 4560 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 21:44:41.229486 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:41.229377 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" podUID="d64f50e0-cf7a-43ec-83ac-a73c0972cb39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 21:44:41.391571 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:41.391525 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:44:42.236769 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:42.236733 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" Oct 30 21:44:42.237974 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:42.237946 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" Oct 30 21:44:42.238800 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:42.238781 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" Oct 30 21:44:42.718163 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:42.718082 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" Oct 30 21:44:42.723830 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:42.723797 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" Oct 30 21:44:42.726593 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:42.726573 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" Oct 30 21:44:42.763878 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:42.763854 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" Oct 30 21:44:42.765011 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:42.764994 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" Oct 30 21:44:42.766083 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:42.766069 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" Oct 30 21:44:43.959367 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:43.958447 4560 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" podUID="4dee11ef-9647-47ee-a4b1-686680256542" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:44:43.982461 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:43.982419 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" podUID="e3281dbe-e8f9-4bab-9ae4-2279e79b7292" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:44:44.128497 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:44.127450 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" podUID="a59a0fc3-8316-47eb-a42e-552d202b1ab7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:44:44.449366 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:44.445420 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" podUID="45a5bac0-28ab-4c78-856e-303f6d4522de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:44:44.504713 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:44.504674 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" podUID="27df5dfe-47a4-4ebb-bd09-1ceef406600a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:44:44.592474 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:44.592435 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" podUID="4dee11ef-9647-47ee-a4b1-686680256542" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Oct 30 21:44:44.629999 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:44.629947 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" podUID="971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:44:44.688599 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:44.686749 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" podUID="90bfb1e4-9526-40f6-aed4-ee46c2d86471" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Oct 30 21:44:44.710409 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:44.708455 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" podUID="8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:44:44.753924 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:44.753873 4560 scope.go:122] "RemoveContainer" containerID="f2b8f0f6398ee28447a619aa3b7e983b36fc263f4f857a611f31c8e605718317" Oct 30 21:44:45.027554 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:45.027488 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" podUID="e3281dbe-e8f9-4bab-9ae4-2279e79b7292" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:44:45.028449 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:45.028426 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" Oct 30 21:44:45.185153 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:45.185124 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" Oct 30 21:44:45.806737 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:45.806702 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc27d458-wwl72"] Oct 30 21:44:45.807225 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:45.807208 4560 cpu_manager.go:414] "containerMap: removing container" podUID="633cda80-9a6a-4366-8dbd-8aa125655e2d" containerName="revoke-keys" Oct 30 21:44:45.807306 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:45.807226 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="633cda80-9a6a-4366-8dbd-8aa125655e2d" containerName="revoke-keys" Oct 30 21:44:45.807377 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:45.807361 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="633cda80-9a6a-4366-8dbd-8aa125655e2d" containerName="revoke-keys" Oct 30 21:44:45.812438 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:45.812418 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc27d458-wwl72" Oct 30 21:44:45.869555 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:45.869520 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w7xd\" (UniqueName: \"kubernetes.io/projected/644973ad-c24e-4e73-bca3-d553a412da80-kube-api-access-6w7xd\") pod \"maas-api-revoke-sub-fc27d458-wwl72\" (UID: \"644973ad-c24e-4e73-bca3-d553a412da80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc27d458-wwl72" Oct 30 21:44:45.869555 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:45.869562 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/644973ad-c24e-4e73-bca3-d553a412da80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc27d458-wwl72\" (UID: \"644973ad-c24e-4e73-bca3-d553a412da80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc27d458-wwl72" Oct 30 21:44:45.869773 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:45.869586 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/644973ad-c24e-4e73-bca3-d553a412da80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc27d458-wwl72\" (UID: \"644973ad-c24e-4e73-bca3-d553a412da80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc27d458-wwl72" Oct 30 21:44:45.882355 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:45.882316 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:44:45.882608 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:45.882583 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:44:45.882709 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:45.882644 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:44:45.884254 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:45.884229 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc27d458-wwl72"] Oct 30 21:44:45.925547 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:45.925526 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:44:45.971264 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:45.971235 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w7xd\" (UniqueName: \"kubernetes.io/projected/644973ad-c24e-4e73-bca3-d553a412da80-kube-api-access-6w7xd\") pod \"maas-api-revoke-sub-fc27d458-wwl72\" (UID: \"644973ad-c24e-4e73-bca3-d553a412da80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc27d458-wwl72" Oct 30 21:44:45.971425 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:45.971271 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/644973ad-c24e-4e73-bca3-d553a412da80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc27d458-wwl72\" (UID: \"644973ad-c24e-4e73-bca3-d553a412da80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc27d458-wwl72" Oct 30 21:44:45.971425 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:45.971292 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/644973ad-c24e-4e73-bca3-d553a412da80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc27d458-wwl72\" (UID: \"644973ad-c24e-4e73-bca3-d553a412da80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc27d458-wwl72" Oct 30 21:44:45.972056 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:45.972034 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/644973ad-c24e-4e73-bca3-d553a412da80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc27d458-wwl72\" (UID: \"644973ad-c24e-4e73-bca3-d553a412da80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc27d458-wwl72" Oct 30 21:44:45.973639 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:45.973617 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/644973ad-c24e-4e73-bca3-d553a412da80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc27d458-wwl72\" (UID: \"644973ad-c24e-4e73-bca3-d553a412da80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc27d458-wwl72" Oct 30 21:44:46.030221 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:46.030192 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w7xd\" (UniqueName: \"kubernetes.io/projected/644973ad-c24e-4e73-bca3-d553a412da80-kube-api-access-6w7xd\") pod \"maas-api-revoke-sub-fc27d458-wwl72\" (UID: \"644973ad-c24e-4e73-bca3-d553a412da80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc27d458-wwl72" Oct 30 21:44:46.125812 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:46.125729 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc27d458-wwl72" Oct 30 21:44:46.513083 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:46.513046 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc27d458-wwl72"] Oct 30 21:44:46.517243 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:44:46.517219 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod644973ad_c24e_4e73_bca3_d553a412da80.slice/crio-3f79e5abfe15b97b16c45e565c9aa4e2d509031bf04be657a6d953b5b03e3b5c WatchSource:0}: Error finding container 3f79e5abfe15b97b16c45e565c9aa4e2d509031bf04be657a6d953b5b03e3b5c: Status 404 returned error can't find the container with id 3f79e5abfe15b97b16c45e565c9aa4e2d509031bf04be657a6d953b5b03e3b5c Oct 30 21:44:46.676012 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:46.675974 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc27d458-wwl72" event={"ID":"644973ad-c24e-4e73-bca3-d553a412da80","Type":"ContainerStarted","Data":"3f79e5abfe15b97b16c45e565c9aa4e2d509031bf04be657a6d953b5b03e3b5c"} Oct 30 21:44:47.684725 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:47.684638 4560 generic.go:356] "Generic (PLEG): container finished" podID="644973ad-c24e-4e73-bca3-d553a412da80" containerID="24ff079eaa0b550cbeec030a978b17af6ee1c29db652091cbc0c9c6c6741722a" exitCode=0 Oct 30 21:44:47.684725 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:47.684710 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc27d458-wwl72" event={"ID":"644973ad-c24e-4e73-bca3-d553a412da80","Type":"ContainerDied","Data":"24ff079eaa0b550cbeec030a978b17af6ee1c29db652091cbc0c9c6c6741722a"} Oct 30 21:44:48.070605 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:48.070574 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f0f1c9b-hqbbz"] Oct 30 21:44:48.075116 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:48.075092 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f0f1c9b-hqbbz"] Oct 30 21:44:48.575981 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:48.575947 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d68e17d8-4657-412e-b1a8-ec1f5b8a04a1" path="/var/lib/kubelet/pods/d68e17d8-4657-412e-b1a8-ec1f5b8a04a1/volumes" Oct 30 21:44:48.844548 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:48.844521 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc27d458-wwl72" Oct 30 21:44:48.912974 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:48.912944 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/644973ad-c24e-4e73-bca3-d553a412da80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/644973ad-c24e-4e73-bca3-d553a412da80-maas-api-openshift-ca\") pod \"644973ad-c24e-4e73-bca3-d553a412da80\" (UID: \"644973ad-c24e-4e73-bca3-d553a412da80\") " Oct 30 21:44:48.913125 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:48.913111 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/644973ad-c24e-4e73-bca3-d553a412da80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/644973ad-c24e-4e73-bca3-d553a412da80-maas-api-xks-ca\") pod \"644973ad-c24e-4e73-bca3-d553a412da80\" (UID: \"644973ad-c24e-4e73-bca3-d553a412da80\") " Oct 30 21:44:48.913201 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:48.913155 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/644973ad-c24e-4e73-bca3-d553a412da80-kube-api-access-6w7xd\" (UniqueName: \"kubernetes.io/projected/644973ad-c24e-4e73-bca3-d553a412da80-kube-api-access-6w7xd\") pod \"644973ad-c24e-4e73-bca3-d553a412da80\" (UID: \"644973ad-c24e-4e73-bca3-d553a412da80\") " Oct 30 21:44:48.913309 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:48.913282 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/644973ad-c24e-4e73-bca3-d553a412da80-maas-api-openshift-ca" pod "644973ad-c24e-4e73-bca3-d553a412da80" (UID: "644973ad-c24e-4e73-bca3-d553a412da80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:48.913598 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:48.913574 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/644973ad-c24e-4e73-bca3-d553a412da80-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:48.914974 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:48.914947 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/644973ad-c24e-4e73-bca3-d553a412da80-maas-api-xks-ca" pod "644973ad-c24e-4e73-bca3-d553a412da80" (UID: "644973ad-c24e-4e73-bca3-d553a412da80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:48.915172 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:48.915155 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/644973ad-c24e-4e73-bca3-d553a412da80-kube-api-access-6w7xd" pod "644973ad-c24e-4e73-bca3-d553a412da80" (UID: "644973ad-c24e-4e73-bca3-d553a412da80"). InnerVolumeSpecName "kube-api-access-6w7xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:49.014857 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:49.014824 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/644973ad-c24e-4e73-bca3-d553a412da80-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:49.014857 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:49.014853 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w7xd\" (UniqueName: \"kubernetes.io/projected/644973ad-c24e-4e73-bca3-d553a412da80-kube-api-access-6w7xd\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:49.684910 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:44:49.684874 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:44:49.699233 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:49.699209 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc27d458-wwl72" Oct 30 21:44:49.699233 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:49.699223 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc27d458-wwl72" event={"ID":"644973ad-c24e-4e73-bca3-d553a412da80","Type":"ContainerDied","Data":"3f79e5abfe15b97b16c45e565c9aa4e2d509031bf04be657a6d953b5b03e3b5c"} Oct 30 21:44:49.699412 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:49.699274 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f79e5abfe15b97b16c45e565c9aa4e2d509031bf04be657a6d953b5b03e3b5c" Oct 30 21:44:50.951646 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:50.951608 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-566bc88f-b98f9"] Oct 30 21:44:50.952505 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:50.952477 4560 cpu_manager.go:414] "containerMap: removing container" podUID="644973ad-c24e-4e73-bca3-d553a412da80" containerName="revoke-keys" Oct 30 21:44:50.952505 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:50.952501 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="644973ad-c24e-4e73-bca3-d553a412da80" containerName="revoke-keys" Oct 30 21:44:50.952677 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:50.952601 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="644973ad-c24e-4e73-bca3-d553a412da80" containerName="revoke-keys" Oct 30 21:44:50.956296 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:50.956277 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-566bc88f-b98f9" Oct 30 21:44:51.037076 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:51.037034 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfrdk\" (UniqueName: \"kubernetes.io/projected/b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7-kube-api-access-pfrdk\") pod \"maas-api-revoke-sub-566bc88f-b98f9\" (UID: \"b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-566bc88f-b98f9" Oct 30 21:44:51.037251 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:51.037086 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-566bc88f-b98f9\" (UID: \"b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-566bc88f-b98f9" Oct 30 21:44:51.037251 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:51.037107 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-566bc88f-b98f9\" (UID: \"b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-566bc88f-b98f9" Oct 30 21:44:51.052817 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:51.052780 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:44:51.052926 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:51.052833 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:44:51.053004 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:51.052987 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:44:51.053052 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:51.053002 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:44:51.056626 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:51.056603 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-566bc88f-b98f9"] Oct 30 21:44:51.138250 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:51.138218 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfrdk\" (UniqueName: \"kubernetes.io/projected/b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7-kube-api-access-pfrdk\") pod \"maas-api-revoke-sub-566bc88f-b98f9\" (UID: \"b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-566bc88f-b98f9" Oct 30 21:44:51.138432 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:51.138275 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-566bc88f-b98f9\" (UID: \"b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-566bc88f-b98f9" Oct 30 21:44:51.138432 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:51.138304 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-566bc88f-b98f9\" (UID: \"b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-566bc88f-b98f9" Oct 30 21:44:51.138945 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:51.138916 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-566bc88f-b98f9\" (UID: \"b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-566bc88f-b98f9" Oct 30 21:44:51.140673 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:51.140654 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-566bc88f-b98f9\" (UID: \"b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-566bc88f-b98f9" Oct 30 21:44:51.198982 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:51.198956 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfrdk\" (UniqueName: \"kubernetes.io/projected/b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7-kube-api-access-pfrdk\") pod \"maas-api-revoke-sub-566bc88f-b98f9\" (UID: \"b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-566bc88f-b98f9" Oct 30 21:44:51.270074 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:51.270041 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-566bc88f-b98f9" Oct 30 21:44:51.645637 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:51.645562 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-566bc88f-b98f9"] Oct 30 21:44:51.650335 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:44:51.650309 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d9d9a7_8bf9_438a_a9fe_5e9ab41150d7.slice/crio-1d4089d7e6e974dfb67c629b67da12acd57c4968ada01e56c79424483d5ec331 WatchSource:0}: Error finding container 1d4089d7e6e974dfb67c629b67da12acd57c4968ada01e56c79424483d5ec331: Status 404 returned error can't find the container with id 1d4089d7e6e974dfb67c629b67da12acd57c4968ada01e56c79424483d5ec331 Oct 30 21:44:51.712662 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:51.712634 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-566bc88f-b98f9" event={"ID":"b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7","Type":"ContainerStarted","Data":"1d4089d7e6e974dfb67c629b67da12acd57c4968ada01e56c79424483d5ec331"} Oct 30 21:44:52.721384 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:52.721318 4560 generic.go:356] "Generic (PLEG): container finished" podID="b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7" containerID="acf7e506cdb282b58ea74ec14dfd77caeca0d7d289dfbb91d98bc22aaa1bb88a" exitCode=0 Oct 30 21:44:52.721849 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:52.721406 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-566bc88f-b98f9" event={"ID":"b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7","Type":"ContainerDied","Data":"acf7e506cdb282b58ea74ec14dfd77caeca0d7d289dfbb91d98bc22aaa1bb88a"} Oct 30 21:44:53.886854 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:53.886830 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-566bc88f-b98f9" Oct 30 21:44:53.980240 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:53.980164 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7-maas-api-xks-ca\") pod \"b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7\" (UID: \"b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7\") " Oct 30 21:44:53.980425 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:53.980402 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7-kube-api-access-pfrdk\" (UniqueName: \"kubernetes.io/projected/b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7-kube-api-access-pfrdk\") pod \"b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7\" (UID: \"b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7\") " Oct 30 21:44:53.980479 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:53.980466 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7-maas-api-openshift-ca\") pod \"b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7\" (UID: \"b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7\") " Oct 30 21:44:53.980767 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:53.980742 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7-maas-api-openshift-ca" pod "b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7" (UID: "b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:53.980950 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:53.980934 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:53.982406 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:53.982375 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7-kube-api-access-pfrdk" pod "b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7" (UID: "b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7"). InnerVolumeSpecName "kube-api-access-pfrdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:53.982533 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:53.982497 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7-maas-api-xks-ca" pod "b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7" (UID: "b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:54.082131 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:54.082100 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:54.082131 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:54.082128 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfrdk\" (UniqueName: \"kubernetes.io/projected/b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7-kube-api-access-pfrdk\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:54.742566 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:54.742533 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-566bc88f-b98f9" Oct 30 21:44:54.742730 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:54.742586 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-566bc88f-b98f9" event={"ID":"b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7","Type":"ContainerDied","Data":"1d4089d7e6e974dfb67c629b67da12acd57c4968ada01e56c79424483d5ec331"} Oct 30 21:44:54.742730 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:54.742654 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d4089d7e6e974dfb67c629b67da12acd57c4968ada01e56c79424483d5ec331" Oct 30 21:44:58.995971 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:58.995939 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b967974-xbnj4"] Oct 30 21:44:58.996711 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:58.996690 4560 cpu_manager.go:414] "containerMap: removing container" podUID="b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7" containerName="revoke-keys" Oct 30 21:44:58.996711 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:58.996712 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7" containerName="revoke-keys" Oct 30 21:44:58.996910 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:58.996896 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7" containerName="revoke-keys" Oct 30 21:44:59.000373 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:59.000353 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b967974-xbnj4" Oct 30 21:44:59.106564 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:59.106530 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:44:59.106718 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:59.106592 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:44:59.106718 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:59.106623 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:44:59.106718 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:59.106669 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:44:59.110438 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:59.110414 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b967974-xbnj4"] Oct 30 21:44:59.138253 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:59.138223 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b2a8f19-1112-42d5-b6cb-f6625301f86e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b967974-xbnj4\" (UID: \"4b2a8f19-1112-42d5-b6cb-f6625301f86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b967974-xbnj4" Oct 30 21:44:59.138407 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:59.138289 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66dzg\" (UniqueName: \"kubernetes.io/projected/4b2a8f19-1112-42d5-b6cb-f6625301f86e-kube-api-access-66dzg\") pod \"maas-api-revoke-sub-9b967974-xbnj4\" (UID: \"4b2a8f19-1112-42d5-b6cb-f6625301f86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b967974-xbnj4" Oct 30 21:44:59.138459 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:59.138442 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b2a8f19-1112-42d5-b6cb-f6625301f86e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b967974-xbnj4\" (UID: \"4b2a8f19-1112-42d5-b6cb-f6625301f86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b967974-xbnj4" Oct 30 21:44:59.239623 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:59.239587 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b2a8f19-1112-42d5-b6cb-f6625301f86e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b967974-xbnj4\" (UID: \"4b2a8f19-1112-42d5-b6cb-f6625301f86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b967974-xbnj4" Oct 30 21:44:59.239807 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:59.239646 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66dzg\" (UniqueName: \"kubernetes.io/projected/4b2a8f19-1112-42d5-b6cb-f6625301f86e-kube-api-access-66dzg\") pod \"maas-api-revoke-sub-9b967974-xbnj4\" (UID: \"4b2a8f19-1112-42d5-b6cb-f6625301f86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b967974-xbnj4" Oct 30 21:44:59.239807 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:59.239716 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b2a8f19-1112-42d5-b6cb-f6625301f86e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b967974-xbnj4\" (UID: \"4b2a8f19-1112-42d5-b6cb-f6625301f86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b967974-xbnj4" Oct 30 21:44:59.240227 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:59.240203 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b2a8f19-1112-42d5-b6cb-f6625301f86e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b967974-xbnj4\" (UID: \"4b2a8f19-1112-42d5-b6cb-f6625301f86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b967974-xbnj4" Oct 30 21:44:59.242001 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:59.241976 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b2a8f19-1112-42d5-b6cb-f6625301f86e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b967974-xbnj4\" (UID: \"4b2a8f19-1112-42d5-b6cb-f6625301f86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b967974-xbnj4" Oct 30 21:44:59.298369 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:59.298294 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66dzg\" (UniqueName: \"kubernetes.io/projected/4b2a8f19-1112-42d5-b6cb-f6625301f86e-kube-api-access-66dzg\") pod \"maas-api-revoke-sub-9b967974-xbnj4\" (UID: \"4b2a8f19-1112-42d5-b6cb-f6625301f86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b967974-xbnj4" Oct 30 21:44:59.313992 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:59.313966 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b967974-xbnj4" Oct 30 21:44:59.718001 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:59.717930 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b967974-xbnj4"] Oct 30 21:44:59.722934 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:44:59.722904 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b2a8f19_1112_42d5_b6cb_f6625301f86e.slice/crio-2431f54d3a1932b3edd9bff34369fddb42cfa84ed2a4824c51790dc56f0e41ad WatchSource:0}: Error finding container 2431f54d3a1932b3edd9bff34369fddb42cfa84ed2a4824c51790dc56f0e41ad: Status 404 returned error can't find the container with id 2431f54d3a1932b3edd9bff34369fddb42cfa84ed2a4824c51790dc56f0e41ad Oct 30 21:44:59.775004 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:44:59.774971 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b967974-xbnj4" event={"ID":"4b2a8f19-1112-42d5-b6cb-f6625301f86e","Type":"ContainerStarted","Data":"2431f54d3a1932b3edd9bff34369fddb42cfa84ed2a4824c51790dc56f0e41ad"} Oct 30 21:45:00.200899 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.200866 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889945-6blmw"] Oct 30 21:45:00.204970 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.204945 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889945-6blmw" Oct 30 21:45:00.208100 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.208074 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a2dced-29889945-w755r"] Oct 30 21:45:00.212661 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.212020 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ws6mz"] Oct 30 21:45:00.212661 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.212132 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a2dced-29889945-w755r" Oct 30 21:45:00.215883 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.215867 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ws6mz" Oct 30 21:45:00.227331 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.227310 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889945-kvfr9"] Oct 30 21:45:00.231177 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.231162 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889945-kvfr9" Oct 30 21:45:00.293571 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.293493 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889945-6blmw"] Oct 30 21:45:00.301379 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.301353 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a2dced-29889945-w755r"] Oct 30 21:45:00.301486 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.301402 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ws6mz"] Oct 30 21:45:00.349469 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.349440 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889945-kvfr9"] Oct 30 21:45:00.353627 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.353604 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j82g\" (UniqueName: \"kubernetes.io/projected/9f301bcc-2a99-4dd3-bddc-b5af224d83f5-kube-api-access-7j82g\") pod \"maas-api-key-cleanup-29889945-ws6mz\" (UID: \"9f301bcc-2a99-4dd3-bddc-b5af224d83f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ws6mz" Oct 30 21:45:00.353727 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.353648 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfk6z\" (UniqueName: \"kubernetes.io/projected/0e9ffebf-af98-439e-82b5-7459d1a6538d-kube-api-access-zfk6z\") pod \"maas-api-key-cleanup-e2e-worker-w4-554f46-29889945-6blmw\" (UID: \"0e9ffebf-af98-439e-82b5-7459d1a6538d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889945-6blmw" Oct 30 21:45:00.353727 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.353669 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqtwv\" (UniqueName: \"kubernetes.io/projected/66765898-e05f-4458-ae55-d69be88bac6a-kube-api-access-qqtwv\") pod \"maas-api-key-cleanup-e2e-worker-w6-a2dced-29889945-w755r\" (UID: \"66765898-e05f-4458-ae55-d69be88bac6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a2dced-29889945-w755r" Oct 30 21:45:00.353858 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.353836 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e9ffebf-af98-439e-82b5-7459d1a6538d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-554f46-29889945-6blmw\" (UID: \"0e9ffebf-af98-439e-82b5-7459d1a6538d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889945-6blmw" Oct 30 21:45:00.353975 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.353957 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b0f43b9-38d3-4e37-a0ce-f987574873b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-71aab5-29889945-kvfr9\" (UID: \"0b0f43b9-38d3-4e37-a0ce-f987574873b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889945-kvfr9" Oct 30 21:45:00.354038 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.354022 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f301bcc-2a99-4dd3-bddc-b5af224d83f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889945-ws6mz\" (UID: \"9f301bcc-2a99-4dd3-bddc-b5af224d83f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ws6mz" Oct 30 21:45:00.354075 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.354063 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66765898-e05f-4458-ae55-d69be88bac6a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-a2dced-29889945-w755r\" (UID: \"66765898-e05f-4458-ae55-d69be88bac6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a2dced-29889945-w755r" Oct 30 21:45:00.354149 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.354137 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htrml\" (UniqueName: \"kubernetes.io/projected/0b0f43b9-38d3-4e37-a0ce-f987574873b9-kube-api-access-htrml\") pod \"maas-api-key-cleanup-e2e-worker-w3-71aab5-29889945-kvfr9\" (UID: \"0b0f43b9-38d3-4e37-a0ce-f987574873b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889945-kvfr9" Oct 30 21:45:00.455218 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.455188 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b0f43b9-38d3-4e37-a0ce-f987574873b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-71aab5-29889945-kvfr9\" (UID: \"0b0f43b9-38d3-4e37-a0ce-f987574873b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889945-kvfr9" Oct 30 21:45:00.455425 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.455241 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f301bcc-2a99-4dd3-bddc-b5af224d83f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889945-ws6mz\" (UID: \"9f301bcc-2a99-4dd3-bddc-b5af224d83f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ws6mz" Oct 30 21:45:00.455425 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.455271 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66765898-e05f-4458-ae55-d69be88bac6a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-a2dced-29889945-w755r\" (UID: \"66765898-e05f-4458-ae55-d69be88bac6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a2dced-29889945-w755r" Oct 30 21:45:00.455425 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.455306 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htrml\" (UniqueName: \"kubernetes.io/projected/0b0f43b9-38d3-4e37-a0ce-f987574873b9-kube-api-access-htrml\") pod \"maas-api-key-cleanup-e2e-worker-w3-71aab5-29889945-kvfr9\" (UID: \"0b0f43b9-38d3-4e37-a0ce-f987574873b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889945-kvfr9" Oct 30 21:45:00.455425 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.455376 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j82g\" (UniqueName: \"kubernetes.io/projected/9f301bcc-2a99-4dd3-bddc-b5af224d83f5-kube-api-access-7j82g\") pod \"maas-api-key-cleanup-29889945-ws6mz\" (UID: \"9f301bcc-2a99-4dd3-bddc-b5af224d83f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ws6mz" Oct 30 21:45:00.455425 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.455405 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfk6z\" (UniqueName: \"kubernetes.io/projected/0e9ffebf-af98-439e-82b5-7459d1a6538d-kube-api-access-zfk6z\") pod \"maas-api-key-cleanup-e2e-worker-w4-554f46-29889945-6blmw\" (UID: \"0e9ffebf-af98-439e-82b5-7459d1a6538d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889945-6blmw" Oct 30 21:45:00.455737 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.455427 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqtwv\" (UniqueName: \"kubernetes.io/projected/66765898-e05f-4458-ae55-d69be88bac6a-kube-api-access-qqtwv\") pod \"maas-api-key-cleanup-e2e-worker-w6-a2dced-29889945-w755r\" (UID: \"66765898-e05f-4458-ae55-d69be88bac6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a2dced-29889945-w755r" Oct 30 21:45:00.455737 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.455492 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e9ffebf-af98-439e-82b5-7459d1a6538d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-554f46-29889945-6blmw\" (UID: \"0e9ffebf-af98-439e-82b5-7459d1a6538d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889945-6blmw" Oct 30 21:45:00.456227 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.456204 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b0f43b9-38d3-4e37-a0ce-f987574873b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-71aab5-29889945-kvfr9\" (UID: \"0b0f43b9-38d3-4e37-a0ce-f987574873b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889945-kvfr9" Oct 30 21:45:00.456227 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.456215 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66765898-e05f-4458-ae55-d69be88bac6a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-a2dced-29889945-w755r\" (UID: \"66765898-e05f-4458-ae55-d69be88bac6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a2dced-29889945-w755r" Oct 30 21:45:00.456392 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.456216 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f301bcc-2a99-4dd3-bddc-b5af224d83f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889945-ws6mz\" (UID: \"9f301bcc-2a99-4dd3-bddc-b5af224d83f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ws6mz" Oct 30 21:45:00.456392 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.456264 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e9ffebf-af98-439e-82b5-7459d1a6538d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-554f46-29889945-6blmw\" (UID: \"0e9ffebf-af98-439e-82b5-7459d1a6538d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889945-6blmw" Oct 30 21:45:00.520986 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.520953 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j82g\" (UniqueName: \"kubernetes.io/projected/9f301bcc-2a99-4dd3-bddc-b5af224d83f5-kube-api-access-7j82g\") pod \"maas-api-key-cleanup-29889945-ws6mz\" (UID: \"9f301bcc-2a99-4dd3-bddc-b5af224d83f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ws6mz" Oct 30 21:45:00.520986 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.520979 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqtwv\" (UniqueName: \"kubernetes.io/projected/66765898-e05f-4458-ae55-d69be88bac6a-kube-api-access-qqtwv\") pod \"maas-api-key-cleanup-e2e-worker-w6-a2dced-29889945-w755r\" (UID: \"66765898-e05f-4458-ae55-d69be88bac6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a2dced-29889945-w755r" Oct 30 21:45:00.522999 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.522973 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfk6z\" (UniqueName: \"kubernetes.io/projected/0e9ffebf-af98-439e-82b5-7459d1a6538d-kube-api-access-zfk6z\") pod \"maas-api-key-cleanup-e2e-worker-w4-554f46-29889945-6blmw\" (UID: \"0e9ffebf-af98-439e-82b5-7459d1a6538d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889945-6blmw" Oct 30 21:45:00.523186 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.523166 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htrml\" (UniqueName: \"kubernetes.io/projected/0b0f43b9-38d3-4e37-a0ce-f987574873b9-kube-api-access-htrml\") pod \"maas-api-key-cleanup-e2e-worker-w3-71aab5-29889945-kvfr9\" (UID: \"0b0f43b9-38d3-4e37-a0ce-f987574873b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889945-kvfr9" Oct 30 21:45:00.527955 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.527936 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a2dced-29889945-w755r" Oct 30 21:45:00.535403 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.535381 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ws6mz" Oct 30 21:45:00.542904 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.542854 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889945-kvfr9" Oct 30 21:45:00.783702 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.783660 4560 generic.go:356] "Generic (PLEG): container finished" podID="4b2a8f19-1112-42d5-b6cb-f6625301f86e" containerID="a8c0e748b87d96333b296539d4b5af68acdaf320812f30cc3ee8ec5aeb74b9dc" exitCode=0 Oct 30 21:45:00.783885 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.783742 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b967974-xbnj4" event={"ID":"4b2a8f19-1112-42d5-b6cb-f6625301f86e","Type":"ContainerDied","Data":"a8c0e748b87d96333b296539d4b5af68acdaf320812f30cc3ee8ec5aeb74b9dc"} Oct 30 21:45:00.819461 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.818559 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889945-6blmw" Oct 30 21:45:00.934394 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.934365 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a2dced-29889945-w755r"] Oct 30 21:45:00.936745 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:45:00.936706 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66765898_e05f_4458_ae55_d69be88bac6a.slice/crio-b7ea6b010ae7c8a262ce513a42d0ce278a287f7413902bb9420f1b9981192cbc WatchSource:0}: Error finding container b7ea6b010ae7c8a262ce513a42d0ce278a287f7413902bb9420f1b9981192cbc: Status 404 returned error can't find the container with id b7ea6b010ae7c8a262ce513a42d0ce278a287f7413902bb9420f1b9981192cbc Oct 30 21:45:00.975977 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.975728 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ws6mz"] Oct 30 21:45:00.979872 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:45:00.979842 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f301bcc_2a99_4dd3_bddc_b5af224d83f5.slice/crio-613facecced298cb098733e4242a91042ce5d4a3f8b31e2ea957d295e6e9653b WatchSource:0}: Error finding container 613facecced298cb098733e4242a91042ce5d4a3f8b31e2ea957d295e6e9653b: Status 404 returned error can't find the container with id 613facecced298cb098733e4242a91042ce5d4a3f8b31e2ea957d295e6e9653b Oct 30 21:45:00.980629 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:00.980601 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889945-kvfr9"] Oct 30 21:45:00.983228 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:45:00.983206 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b0f43b9_38d3_4e37_a0ce_f987574873b9.slice/crio-34df49300348922e6929ae7b052f2fec299f2bcac7ba6c51d5a7ccfc50117ec6 WatchSource:0}: Error finding container 34df49300348922e6929ae7b052f2fec299f2bcac7ba6c51d5a7ccfc50117ec6: Status 404 returned error can't find the container with id 34df49300348922e6929ae7b052f2fec299f2bcac7ba6c51d5a7ccfc50117ec6 Oct 30 21:45:01.201783 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.201755 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889945-6blmw"] Oct 30 21:45:01.205535 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:45:01.205510 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e9ffebf_af98_439e_82b5_7459d1a6538d.slice/crio-760bb281c3368f5e00485224ab74348304418ae98e4fb1e8f4645d6145acfed5 WatchSource:0}: Error finding container 760bb281c3368f5e00485224ab74348304418ae98e4fb1e8f4645d6145acfed5: Status 404 returned error can't find the container with id 760bb281c3368f5e00485224ab74348304418ae98e4fb1e8f4645d6145acfed5 Oct 30 21:45:01.700746 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.700665 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h"] Oct 30 21:45:01.704846 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.704820 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" Oct 30 21:45:01.776908 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.776826 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h"] Oct 30 21:45:01.782014 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.781986 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b"] Oct 30 21:45:01.786658 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.786638 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" Oct 30 21:45:01.794068 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.794038 4560 generic.go:356] "Generic (PLEG): container finished" podID="0b0f43b9-38d3-4e37-a0ce-f987574873b9" containerID="8c94aac07ff142c3dc4ad936ff9ad4aaa48c377295323704607b5c74abd5ce13" exitCode=60 Oct 30 21:45:01.794371 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.794284 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889945-kvfr9" event={"ID":"0b0f43b9-38d3-4e37-a0ce-f987574873b9","Type":"ContainerDied","Data":"8c94aac07ff142c3dc4ad936ff9ad4aaa48c377295323704607b5c74abd5ce13"} Oct 30 21:45:01.794523 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.794376 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889945-kvfr9" event={"ID":"0b0f43b9-38d3-4e37-a0ce-f987574873b9","Type":"ContainerStarted","Data":"34df49300348922e6929ae7b052f2fec299f2bcac7ba6c51d5a7ccfc50117ec6"} Oct 30 21:45:01.795097 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.795079 4560 scope.go:122] "RemoveContainer" containerID="8c94aac07ff142c3dc4ad936ff9ad4aaa48c377295323704607b5c74abd5ce13" Oct 30 21:45:01.797057 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.796962 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889945-6blmw" event={"ID":"0e9ffebf-af98-439e-82b5-7459d1a6538d","Type":"ContainerStarted","Data":"cd55e69aa205f9650d8b863d8565ffffa020916de50ef058bb12a79b99bf513a"} Oct 30 21:45:01.797057 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.797022 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889945-6blmw" event={"ID":"0e9ffebf-af98-439e-82b5-7459d1a6538d","Type":"ContainerStarted","Data":"760bb281c3368f5e00485224ab74348304418ae98e4fb1e8f4645d6145acfed5"} Oct 30 21:45:01.799025 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.798844 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-workload-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h\" (UID: \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" Oct 30 21:45:01.799025 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.798887 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-workload-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b\" (UID: \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" Oct 30 21:45:01.799025 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.798886 4560 generic.go:356] "Generic (PLEG): container finished" podID="66765898-e05f-4458-ae55-d69be88bac6a" containerID="b322e749afb51ae1a5d222a7802aef4fb8795578d5e37c74018e77113666205b" exitCode=60 Oct 30 21:45:01.799025 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.798924 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txkwj\" (UniqueName: \"kubernetes.io/projected/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-kube-api-access-txkwj\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h\" (UID: \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" Oct 30 21:45:01.799025 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.798953 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-istio-token\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h\" (UID: \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" Oct 30 21:45:01.799025 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.798992 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-istio-data\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h\" (UID: \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" Oct 30 21:45:01.799025 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.799001 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a2dced-29889945-w755r" event={"ID":"66765898-e05f-4458-ae55-d69be88bac6a","Type":"ContainerDied","Data":"b322e749afb51ae1a5d222a7802aef4fb8795578d5e37c74018e77113666205b"} Oct 30 21:45:01.799025 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.799022 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-credential-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b\" (UID: \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" Oct 30 21:45:01.799487 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.799035 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a2dced-29889945-w755r" event={"ID":"66765898-e05f-4458-ae55-d69be88bac6a","Type":"ContainerStarted","Data":"b7ea6b010ae7c8a262ce513a42d0ce278a287f7413902bb9420f1b9981192cbc"} Oct 30 21:45:01.799487 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.799046 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-istio-data\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b\" (UID: \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" Oct 30 21:45:01.799487 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.799089 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-workload-certs\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b\" (UID: \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" Oct 30 21:45:01.799487 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.799131 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-workload-certs\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h\" (UID: \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" Oct 30 21:45:01.799487 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.799156 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khqbx\" (UniqueName: \"kubernetes.io/projected/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-kube-api-access-khqbx\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b\" (UID: \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" Oct 30 21:45:01.799487 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.799279 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-istio-envoy\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b\" (UID: \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" Oct 30 21:45:01.799487 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.799290 4560 scope.go:122] "RemoveContainer" containerID="b322e749afb51ae1a5d222a7802aef4fb8795578d5e37c74018e77113666205b" Oct 30 21:45:01.799487 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.799307 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-istio-envoy\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h\" (UID: \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" Oct 30 21:45:01.799487 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.799331 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-istio-podinfo\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h\" (UID: \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" Oct 30 21:45:01.799487 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.799407 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-istio-token\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b\" (UID: \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" Oct 30 21:45:01.799487 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.799446 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-istiod-ca-cert\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b\" (UID: \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" Oct 30 21:45:01.799487 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.799469 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-istiod-ca-cert\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h\" (UID: \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" Oct 30 21:45:01.800036 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.799510 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-credential-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h\" (UID: \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" Oct 30 21:45:01.800036 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.799531 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-istio-podinfo\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b\" (UID: \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" Oct 30 21:45:01.802488 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.801675 4560 generic.go:356] "Generic (PLEG): container finished" podID="9f301bcc-2a99-4dd3-bddc-b5af224d83f5" containerID="f20fed83f69f8dd87fb02d32923bb8642033461dd032e4bd2f4381eadae2156e" exitCode=60 Oct 30 21:45:01.802488 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.802104 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ws6mz" event={"ID":"9f301bcc-2a99-4dd3-bddc-b5af224d83f5","Type":"ContainerDied","Data":"f20fed83f69f8dd87fb02d32923bb8642033461dd032e4bd2f4381eadae2156e"} Oct 30 21:45:01.802488 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.802164 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ws6mz" event={"ID":"9f301bcc-2a99-4dd3-bddc-b5af224d83f5","Type":"ContainerStarted","Data":"613facecced298cb098733e4242a91042ce5d4a3f8b31e2ea957d295e6e9653b"} Oct 30 21:45:01.802488 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.802195 4560 scope.go:122] "RemoveContainer" containerID="f20fed83f69f8dd87fb02d32923bb8642033461dd032e4bd2f4381eadae2156e" Oct 30 21:45:01.845801 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.845760 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b"] Oct 30 21:45:01.900752 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.900722 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-workload-certs\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h\" (UID: \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" Oct 30 21:45:01.900881 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.900767 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khqbx\" (UniqueName: \"kubernetes.io/projected/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-kube-api-access-khqbx\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b\" (UID: \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" Oct 30 21:45:01.900946 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.900891 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-istio-envoy\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b\" (UID: \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" Oct 30 21:45:01.900946 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.900919 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-istio-envoy\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h\" (UID: \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" Oct 30 21:45:01.901054 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.900947 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-istio-podinfo\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h\" (UID: \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" Oct 30 21:45:01.901054 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.900976 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-istio-token\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b\" (UID: \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" Oct 30 21:45:01.901054 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.901011 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-istiod-ca-cert\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b\" (UID: \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" Oct 30 21:45:01.901054 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.901037 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-istiod-ca-cert\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h\" (UID: \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" Oct 30 21:45:01.901261 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.901077 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-credential-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h\" (UID: \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" Oct 30 21:45:01.901261 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.901101 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-istio-podinfo\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b\" (UID: \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" Oct 30 21:45:01.901261 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.901221 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-workload-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h\" (UID: \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" Oct 30 21:45:01.901436 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.901307 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-workload-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b\" (UID: \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" Oct 30 21:45:01.901436 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.901372 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txkwj\" (UniqueName: \"kubernetes.io/projected/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-kube-api-access-txkwj\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h\" (UID: \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" Oct 30 21:45:01.901436 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.901411 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-istio-token\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h\" (UID: \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" Oct 30 21:45:01.901586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.901466 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-istio-data\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h\" (UID: \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" Oct 30 21:45:01.901586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.901504 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-credential-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b\" (UID: \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" Oct 30 21:45:01.901586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.901527 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-istio-data\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b\" (UID: \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" Oct 30 21:45:01.901740 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.901582 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-workload-certs\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b\" (UID: \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" Oct 30 21:45:01.901924 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.901893 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-istiod-ca-cert\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b\" (UID: \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" Oct 30 21:45:01.901997 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.901936 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-workload-certs\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b\" (UID: \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" Oct 30 21:45:01.904969 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.902140 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-istiod-ca-cert\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h\" (UID: \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" Oct 30 21:45:01.904969 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.902203 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-workload-certs\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h\" (UID: \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" Oct 30 21:45:01.904969 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.902479 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-credential-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h\" (UID: \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" Oct 30 21:45:01.904969 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.902956 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-credential-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b\" (UID: \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" Oct 30 21:45:01.904969 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.903166 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-workload-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h\" (UID: \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" Oct 30 21:45:01.904969 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.903192 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-istio-data\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h\" (UID: \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" Oct 30 21:45:01.904969 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.903398 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-istio-data\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b\" (UID: \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" Oct 30 21:45:01.904969 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.903438 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-workload-socket\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b\" (UID: \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" Oct 30 21:45:01.905480 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.905056 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-istio-podinfo\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b\" (UID: \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" Oct 30 21:45:01.905913 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.905733 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-istio-envoy\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h\" (UID: \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" Oct 30 21:45:01.907580 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.907553 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-istio-envoy\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b\" (UID: \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" Oct 30 21:45:01.907793 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:01.907766 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-istio-podinfo\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h\" (UID: \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" Oct 30 21:45:02.028100 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.028058 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-istio-token\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h\" (UID: \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" Oct 30 21:45:02.028613 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.028582 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-istio-token\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b\" (UID: \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" Oct 30 21:45:02.028788 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.028767 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khqbx\" (UniqueName: \"kubernetes.io/projected/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-kube-api-access-khqbx\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b\" (UID: \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" Oct 30 21:45:02.029026 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.028996 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txkwj\" (UniqueName: \"kubernetes.io/projected/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-kube-api-access-txkwj\") pod \"e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h\" (UID: \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\") " pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" Oct 30 21:45:02.049924 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.049897 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b967974-xbnj4" Oct 30 21:45:02.054509 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.054485 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a7b4e06-xv9vg"] Oct 30 21:45:02.059028 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.059003 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a7b4e06-xv9vg"] Oct 30 21:45:02.075314 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.075262 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889945-6blmw" podStartSLOduration=2.075249916 podStartE2EDuration="2.075249916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:02.071814079 +0000 UTC m=+1877.626513855" watchObservedRunningTime="2026-10-30 21:45:02.075249916 +0000 UTC m=+1877.629949692" Oct 30 21:45:02.103277 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.103249 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b2a8f19-1112-42d5-b6cb-f6625301f86e-kube-api-access-66dzg\" (UniqueName: \"kubernetes.io/projected/4b2a8f19-1112-42d5-b6cb-f6625301f86e-kube-api-access-66dzg\") pod \"4b2a8f19-1112-42d5-b6cb-f6625301f86e\" (UID: \"4b2a8f19-1112-42d5-b6cb-f6625301f86e\") " Oct 30 21:45:02.103404 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.103291 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b2a8f19-1112-42d5-b6cb-f6625301f86e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b2a8f19-1112-42d5-b6cb-f6625301f86e-maas-api-xks-ca\") pod \"4b2a8f19-1112-42d5-b6cb-f6625301f86e\" (UID: \"4b2a8f19-1112-42d5-b6cb-f6625301f86e\") " Oct 30 21:45:02.103447 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.103408 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b2a8f19-1112-42d5-b6cb-f6625301f86e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b2a8f19-1112-42d5-b6cb-f6625301f86e-maas-api-openshift-ca\") pod \"4b2a8f19-1112-42d5-b6cb-f6625301f86e\" (UID: \"4b2a8f19-1112-42d5-b6cb-f6625301f86e\") " Oct 30 21:45:02.103816 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.103791 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b2a8f19-1112-42d5-b6cb-f6625301f86e-maas-api-openshift-ca" pod "4b2a8f19-1112-42d5-b6cb-f6625301f86e" (UID: "4b2a8f19-1112-42d5-b6cb-f6625301f86e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:02.103954 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.103940 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b2a8f19-1112-42d5-b6cb-f6625301f86e-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:02.128975 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.128945 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b2a8f19-1112-42d5-b6cb-f6625301f86e-maas-api-xks-ca" pod "4b2a8f19-1112-42d5-b6cb-f6625301f86e" (UID: "4b2a8f19-1112-42d5-b6cb-f6625301f86e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:02.129174 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.129150 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b2a8f19-1112-42d5-b6cb-f6625301f86e-kube-api-access-66dzg" pod "4b2a8f19-1112-42d5-b6cb-f6625301f86e" (UID: "4b2a8f19-1112-42d5-b6cb-f6625301f86e"). InnerVolumeSpecName "kube-api-access-66dzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:02.205006 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.204933 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66dzg\" (UniqueName: \"kubernetes.io/projected/4b2a8f19-1112-42d5-b6cb-f6625301f86e-kube-api-access-66dzg\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:02.205006 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.204960 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b2a8f19-1112-42d5-b6cb-f6625301f86e-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:02.267173 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.267146 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" Oct 30 21:45:02.318887 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.318852 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" Oct 30 21:45:02.577906 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.577869 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d30312af-f1f2-40e0-b630-db46b93837eb" path="/var/lib/kubelet/pods/d30312af-f1f2-40e0-b630-db46b93837eb/volumes" Oct 30 21:45:02.659618 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.659581 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b"] Oct 30 21:45:02.662937 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:45:02.662911 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e1ac4c8_c9d4_4419_8a37_cabedad5ad01.slice/crio-3c1d9964e501511a4953c70a9f94d4d4a17c59e1bd9484db90ced20fc393298f WatchSource:0}: Error finding container 3c1d9964e501511a4953c70a9f94d4d4a17c59e1bd9484db90ced20fc393298f: Status 404 returned error can't find the container with id 3c1d9964e501511a4953c70a9f94d4d4a17c59e1bd9484db90ced20fc393298f Oct 30 21:45:02.665785 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.665744 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:45:02.665912 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.665830 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:45:02.665912 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.665869 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:45:02.736495 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.736468 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h"] Oct 30 21:45:02.786569 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:45:02.786527 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode71b8a5c_fcd2_44f0_86d4_984ab9512afd.slice/crio-0c326e63d01ee847d2bfd83aee127cb7972194703f13d276ac7b7485f109b4bb WatchSource:0}: Error finding container 0c326e63d01ee847d2bfd83aee127cb7972194703f13d276ac7b7485f109b4bb: Status 404 returned error can't find the container with id 0c326e63d01ee847d2bfd83aee127cb7972194703f13d276ac7b7485f109b4bb Oct 30 21:45:02.789406 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.789372 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:45:02.789505 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.789441 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:45:02.789505 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.789468 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:45:02.816936 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.816906 4560 generic.go:356] "Generic (PLEG): container finished" podID="0e9ffebf-af98-439e-82b5-7459d1a6538d" containerID="cd55e69aa205f9650d8b863d8565ffffa020916de50ef058bb12a79b99bf513a" exitCode=60 Oct 30 21:45:02.817066 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.816990 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889945-6blmw" event={"ID":"0e9ffebf-af98-439e-82b5-7459d1a6538d","Type":"ContainerDied","Data":"cd55e69aa205f9650d8b863d8565ffffa020916de50ef058bb12a79b99bf513a"} Oct 30 21:45:02.817290 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.817273 4560 scope.go:122] "RemoveContainer" containerID="cd55e69aa205f9650d8b863d8565ffffa020916de50ef058bb12a79b99bf513a" Oct 30 21:45:02.818976 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.818952 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" event={"ID":"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01","Type":"ContainerStarted","Data":"2713788d053dd37274192bceae58790d357dc83ecdb7f696140b4ebded3d85da"} Oct 30 21:45:02.819082 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.819002 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" event={"ID":"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01","Type":"ContainerStarted","Data":"3c1d9964e501511a4953c70a9f94d4d4a17c59e1bd9484db90ced20fc393298f"} Oct 30 21:45:02.820400 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.820381 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" event={"ID":"e71b8a5c-fcd2-44f0-86d4-984ab9512afd","Type":"ContainerStarted","Data":"0c326e63d01ee847d2bfd83aee127cb7972194703f13d276ac7b7485f109b4bb"} Oct 30 21:45:02.822550 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.822526 4560 generic.go:356] "Generic (PLEG): container finished" podID="66765898-e05f-4458-ae55-d69be88bac6a" containerID="06c06e1b62c9ed7a5d285f515bd5a5cefeb290c2f213b09b4b08c54c29420ae5" exitCode=60 Oct 30 21:45:02.822660 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.822607 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a2dced-29889945-w755r" event={"ID":"66765898-e05f-4458-ae55-d69be88bac6a","Type":"ContainerDied","Data":"06c06e1b62c9ed7a5d285f515bd5a5cefeb290c2f213b09b4b08c54c29420ae5"} Oct 30 21:45:02.822660 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.822647 4560 scope.go:122] "RemoveContainer" containerID="b322e749afb51ae1a5d222a7802aef4fb8795578d5e37c74018e77113666205b" Oct 30 21:45:02.823020 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.823004 4560 scope.go:122] "RemoveContainer" containerID="06c06e1b62c9ed7a5d285f515bd5a5cefeb290c2f213b09b4b08c54c29420ae5" Oct 30 21:45:02.823295 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:45:02.823227 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-a2dced-29889945-w755r_odh-ai-gateway-infra(66765898-e05f-4458-ae55-d69be88bac6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a2dced-29889945-w755r" podUID="66765898-e05f-4458-ae55-d69be88bac6a" Oct 30 21:45:02.825332 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.825297 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b967974-xbnj4" event={"ID":"4b2a8f19-1112-42d5-b6cb-f6625301f86e","Type":"ContainerDied","Data":"2431f54d3a1932b3edd9bff34369fddb42cfa84ed2a4824c51790dc56f0e41ad"} Oct 30 21:45:02.825488 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.825471 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b967974-xbnj4" Oct 30 21:45:02.828102 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.828020 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2431f54d3a1932b3edd9bff34369fddb42cfa84ed2a4824c51790dc56f0e41ad" Oct 30 21:45:02.835100 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.835071 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ws6mz" event={"ID":"9f301bcc-2a99-4dd3-bddc-b5af224d83f5","Type":"ContainerStarted","Data":"42854d8f5b5bcbb431db8f390e2032edb55eb53e6fdfc184f757e451eeac0300"} Oct 30 21:45:02.841064 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:02.841040 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889945-kvfr9" event={"ID":"0b0f43b9-38d3-4e37-a0ce-f987574873b9","Type":"ContainerStarted","Data":"7af3290e705b277167391a217fde730e53daff69165b2068c20c3c256afd275c"} Oct 30 21:45:03.050181 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:03.050110 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" podStartSLOduration=2.050093168 podStartE2EDuration="2.050093168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:03.049543405 +0000 UTC m=+1878.604243193" watchObservedRunningTime="2026-10-30 21:45:03.050093168 +0000 UTC m=+1878.604792946" Oct 30 21:45:03.165684 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:03.165621 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889945-kvfr9" podStartSLOduration=3.165602575 podStartE2EDuration="3.165602575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:03.160945617 +0000 UTC m=+1878.715645395" watchObservedRunningTime="2026-10-30 21:45:03.165602575 +0000 UTC m=+1878.720302355" Oct 30 21:45:03.267833 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:03.267802 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" Oct 30 21:45:03.269150 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:03.269126 4560 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 21:45:03.269241 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:03.269179 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" podUID="8e1ac4c8-c9d4-4419-8a37-cabedad5ad01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 21:45:03.284987 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:03.284927 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ws6mz" podStartSLOduration=3.284910755 podStartE2EDuration="3.284910755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:03.284385055 +0000 UTC m=+1878.839084831" watchObservedRunningTime="2026-10-30 21:45:03.284910755 +0000 UTC m=+1878.839610533" Oct 30 21:45:03.880278 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:03.880248 4560 generic.go:356] "Generic (PLEG): container finished" podID="9f301bcc-2a99-4dd3-bddc-b5af224d83f5" containerID="42854d8f5b5bcbb431db8f390e2032edb55eb53e6fdfc184f757e451eeac0300" exitCode=60 Oct 30 21:45:03.880934 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:03.880385 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ws6mz" event={"ID":"9f301bcc-2a99-4dd3-bddc-b5af224d83f5","Type":"ContainerDied","Data":"42854d8f5b5bcbb431db8f390e2032edb55eb53e6fdfc184f757e451eeac0300"} Oct 30 21:45:03.881097 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:03.880861 4560 scope.go:122] "RemoveContainer" containerID="42854d8f5b5bcbb431db8f390e2032edb55eb53e6fdfc184f757e451eeac0300" Oct 30 21:45:03.881157 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:03.881138 4560 scope.go:122] "RemoveContainer" containerID="f20fed83f69f8dd87fb02d32923bb8642033461dd032e4bd2f4381eadae2156e" Oct 30 21:45:03.881553 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:45:03.881528 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889945-ws6mz_odh-ai-gateway-infra(9f301bcc-2a99-4dd3-bddc-b5af224d83f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ws6mz" podUID="9f301bcc-2a99-4dd3-bddc-b5af224d83f5" Oct 30 21:45:03.883938 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:03.883914 4560 generic.go:356] "Generic (PLEG): container finished" podID="0b0f43b9-38d3-4e37-a0ce-f987574873b9" containerID="7af3290e705b277167391a217fde730e53daff69165b2068c20c3c256afd275c" exitCode=60 Oct 30 21:45:03.884077 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:03.884057 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889945-kvfr9" event={"ID":"0b0f43b9-38d3-4e37-a0ce-f987574873b9","Type":"ContainerDied","Data":"7af3290e705b277167391a217fde730e53daff69165b2068c20c3c256afd275c"} Oct 30 21:45:03.884282 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:03.884267 4560 scope.go:122] "RemoveContainer" containerID="7af3290e705b277167391a217fde730e53daff69165b2068c20c3c256afd275c" Oct 30 21:45:03.885728 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:45:03.885703 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-71aab5-29889945-kvfr9_odh-ai-gateway-infra(0b0f43b9-38d3-4e37-a0ce-f987574873b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889945-kvfr9" podUID="0b0f43b9-38d3-4e37-a0ce-f987574873b9" Oct 30 21:45:03.886480 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:03.886459 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889945-6blmw" event={"ID":"0e9ffebf-af98-439e-82b5-7459d1a6538d","Type":"ContainerStarted","Data":"7f93f3d2d251ed0f67f962440be02b170ef3885c94d68b335db470848090b2f3"} Oct 30 21:45:03.888512 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:03.888487 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" event={"ID":"e71b8a5c-fcd2-44f0-86d4-984ab9512afd","Type":"ContainerStarted","Data":"000119b73bc1afe20c21ac31375dae42bc0e428028d3389b510571d8568929e7"} Oct 30 21:45:03.890682 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:03.890661 4560 scope.go:122] "RemoveContainer" containerID="06c06e1b62c9ed7a5d285f515bd5a5cefeb290c2f213b09b4b08c54c29420ae5" Oct 30 21:45:03.890833 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:45:03.890814 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-a2dced-29889945-w755r_odh-ai-gateway-infra(66765898-e05f-4458-ae55-d69be88bac6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a2dced-29889945-w755r" podUID="66765898-e05f-4458-ae55-d69be88bac6a" Oct 30 21:45:03.895194 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:03.895168 4560 scope.go:122] "RemoveContainer" containerID="8c94aac07ff142c3dc4ad936ff9ad4aaa48c377295323704607b5c74abd5ce13" Oct 30 21:45:04.270364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:04.267965 4560 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 21:45:04.270364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:04.268044 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" podUID="8e1ac4c8-c9d4-4419-8a37-cabedad5ad01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 21:45:04.319530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:04.319500 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" Oct 30 21:45:04.320958 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:04.320934 4560 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:45:04.321076 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:04.320977 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" podUID="e71b8a5c-fcd2-44f0-86d4-984ab9512afd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 21:45:04.385806 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:04.385747 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" podStartSLOduration=3.385732935 podStartE2EDuration="3.385732935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:04.381787151 +0000 UTC m=+1879.936486931" watchObservedRunningTime="2026-10-30 21:45:04.385732935 +0000 UTC m=+1879.940432706" Oct 30 21:45:04.898300 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:04.898265 4560 generic.go:356] "Generic (PLEG): container finished" podID="0e9ffebf-af98-439e-82b5-7459d1a6538d" containerID="7f93f3d2d251ed0f67f962440be02b170ef3885c94d68b335db470848090b2f3" exitCode=60 Oct 30 21:45:04.898485 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:04.898362 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889945-6blmw" event={"ID":"0e9ffebf-af98-439e-82b5-7459d1a6538d","Type":"ContainerDied","Data":"7f93f3d2d251ed0f67f962440be02b170ef3885c94d68b335db470848090b2f3"} Oct 30 21:45:04.898485 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:04.898457 4560 scope.go:122] "RemoveContainer" containerID="cd55e69aa205f9650d8b863d8565ffffa020916de50ef058bb12a79b99bf513a" Oct 30 21:45:04.898621 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:04.898603 4560 scope.go:122] "RemoveContainer" containerID="7f93f3d2d251ed0f67f962440be02b170ef3885c94d68b335db470848090b2f3" Oct 30 21:45:04.898948 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:45:04.898923 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-554f46-29889945-6blmw_odh-ai-gateway-infra(0e9ffebf-af98-439e-82b5-7459d1a6538d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889945-6blmw" podUID="0e9ffebf-af98-439e-82b5-7459d1a6538d" Oct 30 21:45:04.900890 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:04.900869 4560 scope.go:122] "RemoveContainer" containerID="42854d8f5b5bcbb431db8f390e2032edb55eb53e6fdfc184f757e451eeac0300" Oct 30 21:45:04.901160 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:45:04.901140 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889945-ws6mz_odh-ai-gateway-infra(9f301bcc-2a99-4dd3-bddc-b5af224d83f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ws6mz" podUID="9f301bcc-2a99-4dd3-bddc-b5af224d83f5" Oct 30 21:45:04.902265 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:04.902248 4560 scope.go:122] "RemoveContainer" containerID="7af3290e705b277167391a217fde730e53daff69165b2068c20c3c256afd275c" Oct 30 21:45:04.902482 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:45:04.902465 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-71aab5-29889945-kvfr9_odh-ai-gateway-infra(0b0f43b9-38d3-4e37-a0ce-f987574873b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889945-kvfr9" podUID="0b0f43b9-38d3-4e37-a0ce-f987574873b9" Oct 30 21:45:05.079490 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:05.079442 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03a645de-426dc"] Oct 30 21:45:05.085017 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:05.084989 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03a645de-426dc"] Oct 30 21:45:05.279105 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:05.278608 4560 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 21:45:05.279105 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:05.278667 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" podUID="8e1ac4c8-c9d4-4419-8a37-cabedad5ad01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 21:45:05.319311 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:05.319270 4560 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:45:05.319487 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:05.319321 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" podUID="e71b8a5c-fcd2-44f0-86d4-984ab9512afd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 21:45:05.990711 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:05.990684 4560 scope.go:122] "RemoveContainer" containerID="7f93f3d2d251ed0f67f962440be02b170ef3885c94d68b335db470848090b2f3" Oct 30 21:45:05.990949 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:45:05.990926 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-554f46-29889945-6blmw_odh-ai-gateway-infra(0e9ffebf-af98-439e-82b5-7459d1a6538d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889945-6blmw" podUID="0e9ffebf-af98-439e-82b5-7459d1a6538d" Oct 30 21:45:06.273421 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:06.272833 4560 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 21:45:06.273421 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:06.272897 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" podUID="8e1ac4c8-c9d4-4419-8a37-cabedad5ad01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 21:45:06.319649 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:06.319620 4560 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:45:06.320145 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:06.320118 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" podUID="e71b8a5c-fcd2-44f0-86d4-984ab9512afd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 21:45:06.585401 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:06.585003 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42451875-993c-4186-a4a6-fad56871700a" path="/var/lib/kubelet/pods/42451875-993c-4186-a4a6-fad56871700a/volumes" Oct 30 21:45:06.756362 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:06.756223 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-637696f1-pvfq6"] Oct 30 21:45:06.760360 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:06.757048 4560 cpu_manager.go:414] "containerMap: removing container" podUID="4b2a8f19-1112-42d5-b6cb-f6625301f86e" containerName="revoke-keys" Oct 30 21:45:06.760360 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:06.757075 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b2a8f19-1112-42d5-b6cb-f6625301f86e" containerName="revoke-keys" Oct 30 21:45:06.760360 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:06.757235 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b2a8f19-1112-42d5-b6cb-f6625301f86e" containerName="revoke-keys" Oct 30 21:45:06.781369 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:06.778108 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637696f1-pvfq6" Oct 30 21:45:06.800580 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:06.800553 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-637696f1-pvfq6"] Oct 30 21:45:06.813504 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:06.813174 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-637696f1-pvfq6\" (UID: \"cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637696f1-pvfq6" Oct 30 21:45:06.813504 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:06.813287 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-637696f1-pvfq6\" (UID: \"cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637696f1-pvfq6" Oct 30 21:45:06.813504 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:06.813403 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmjj6\" (UniqueName: \"kubernetes.io/projected/cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c-kube-api-access-jmjj6\") pod \"maas-api-revoke-sub-637696f1-pvfq6\" (UID: \"cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637696f1-pvfq6" Oct 30 21:45:06.834362 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:06.833072 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:45:06.914293 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:06.914203 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-637696f1-pvfq6\" (UID: \"cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637696f1-pvfq6" Oct 30 21:45:06.914293 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:06.914290 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmjj6\" (UniqueName: \"kubernetes.io/projected/cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c-kube-api-access-jmjj6\") pod \"maas-api-revoke-sub-637696f1-pvfq6\" (UID: \"cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637696f1-pvfq6" Oct 30 21:45:06.914553 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:06.914398 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-637696f1-pvfq6\" (UID: \"cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637696f1-pvfq6" Oct 30 21:45:06.914906 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:06.914881 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-637696f1-pvfq6\" (UID: \"cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637696f1-pvfq6" Oct 30 21:45:06.916729 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:06.916708 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-637696f1-pvfq6\" (UID: \"cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637696f1-pvfq6" Oct 30 21:45:06.980198 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:06.980161 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmjj6\" (UniqueName: \"kubernetes.io/projected/cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c-kube-api-access-jmjj6\") pod \"maas-api-revoke-sub-637696f1-pvfq6\" (UID: \"cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637696f1-pvfq6" Oct 30 21:45:07.097114 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:07.097078 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637696f1-pvfq6" Oct 30 21:45:07.281920 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:07.281867 4560 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 21:45:07.282100 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:07.281942 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" podUID="8e1ac4c8-c9d4-4419-8a37-cabedad5ad01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 21:45:07.320982 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:07.320936 4560 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:45:07.321484 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:07.321002 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" podUID="e71b8a5c-fcd2-44f0-86d4-984ab9512afd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 21:45:07.550366 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:07.549803 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-637696f1-pvfq6"] Oct 30 21:45:07.577361 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:45:07.574777 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf67bd40_7ae8_45c7_97f3_3e89f2ddef5c.slice/crio-33ae934b99378317ee985e05f30721ddefdbfddaa9412fc6b8b4c0c126fc2197 WatchSource:0}: Error finding container 33ae934b99378317ee985e05f30721ddefdbfddaa9412fc6b8b4c0c126fc2197: Status 404 returned error can't find the container with id 33ae934b99378317ee985e05f30721ddefdbfddaa9412fc6b8b4c0c126fc2197 Oct 30 21:45:08.005250 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:08.005221 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637696f1-pvfq6" event={"ID":"cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c","Type":"ContainerStarted","Data":"964755de842d9cc9e9e74e310e3ea127ac02f60b0127cfe298d72f1b31061497"} Oct 30 21:45:08.005377 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:08.005272 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637696f1-pvfq6" event={"ID":"cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c","Type":"ContainerStarted","Data":"33ae934b99378317ee985e05f30721ddefdbfddaa9412fc6b8b4c0c126fc2197"} Oct 30 21:45:08.124860 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:08.124803 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637696f1-pvfq6" podStartSLOduration=2.124790193 podStartE2EDuration="2.124790193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:08.122838892 +0000 UTC m=+1883.677538671" watchObservedRunningTime="2026-10-30 21:45:08.124790193 +0000 UTC m=+1883.679489970" Oct 30 21:45:08.273897 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:08.273801 4560 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 21:45:08.274162 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:08.274134 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" podUID="8e1ac4c8-c9d4-4419-8a37-cabedad5ad01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 21:45:08.320165 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:08.320061 4560 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:45:08.320165 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:08.320122 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" podUID="e71b8a5c-fcd2-44f0-86d4-984ab9512afd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 21:45:09.016108 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:09.016069 4560 generic.go:356] "Generic (PLEG): container finished" podID="cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c" containerID="964755de842d9cc9e9e74e310e3ea127ac02f60b0127cfe298d72f1b31061497" exitCode=0 Oct 30 21:45:09.016534 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:09.016151 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637696f1-pvfq6" event={"ID":"cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c","Type":"ContainerDied","Data":"964755de842d9cc9e9e74e310e3ea127ac02f60b0127cfe298d72f1b31061497"} Oct 30 21:45:09.267671 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:09.267586 4560 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 21:45:09.267671 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:09.267644 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" podUID="8e1ac4c8-c9d4-4419-8a37-cabedad5ad01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 21:45:09.269663 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:09.269640 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a01ea9f-2pcg5"] Oct 30 21:45:09.273622 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:09.273602 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a01ea9f-2pcg5" Oct 30 21:45:09.320257 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:09.320223 4560 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:45:09.320416 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:09.320295 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" podUID="e71b8a5c-fcd2-44f0-86d4-984ab9512afd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 21:45:09.335036 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:09.335005 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a01ea9f-2pcg5"] Oct 30 21:45:09.391017 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:09.390967 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2efef570-20ac-4b9e-9d4e-f78bc752c0a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a01ea9f-2pcg5\" (UID: \"2efef570-20ac-4b9e-9d4e-f78bc752c0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a01ea9f-2pcg5" Oct 30 21:45:09.391221 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:09.391025 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2efef570-20ac-4b9e-9d4e-f78bc752c0a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a01ea9f-2pcg5\" (UID: \"2efef570-20ac-4b9e-9d4e-f78bc752c0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a01ea9f-2pcg5" Oct 30 21:45:09.391402 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:09.391374 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw85n\" (UniqueName: \"kubernetes.io/projected/2efef570-20ac-4b9e-9d4e-f78bc752c0a1-kube-api-access-nw85n\") pod \"maas-api-revoke-sub-3a01ea9f-2pcg5\" (UID: \"2efef570-20ac-4b9e-9d4e-f78bc752c0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a01ea9f-2pcg5" Oct 30 21:45:09.492895 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:09.492865 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2efef570-20ac-4b9e-9d4e-f78bc752c0a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a01ea9f-2pcg5\" (UID: \"2efef570-20ac-4b9e-9d4e-f78bc752c0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a01ea9f-2pcg5" Oct 30 21:45:09.493065 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:09.492902 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2efef570-20ac-4b9e-9d4e-f78bc752c0a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a01ea9f-2pcg5\" (UID: \"2efef570-20ac-4b9e-9d4e-f78bc752c0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a01ea9f-2pcg5" Oct 30 21:45:09.493145 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:09.493089 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw85n\" (UniqueName: \"kubernetes.io/projected/2efef570-20ac-4b9e-9d4e-f78bc752c0a1-kube-api-access-nw85n\") pod \"maas-api-revoke-sub-3a01ea9f-2pcg5\" (UID: \"2efef570-20ac-4b9e-9d4e-f78bc752c0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a01ea9f-2pcg5" Oct 30 21:45:09.493849 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:09.493817 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2efef570-20ac-4b9e-9d4e-f78bc752c0a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a01ea9f-2pcg5\" (UID: \"2efef570-20ac-4b9e-9d4e-f78bc752c0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a01ea9f-2pcg5" Oct 30 21:45:09.495666 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:09.495626 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2efef570-20ac-4b9e-9d4e-f78bc752c0a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a01ea9f-2pcg5\" (UID: \"2efef570-20ac-4b9e-9d4e-f78bc752c0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a01ea9f-2pcg5" Oct 30 21:45:09.554452 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:09.554383 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw85n\" (UniqueName: \"kubernetes.io/projected/2efef570-20ac-4b9e-9d4e-f78bc752c0a1-kube-api-access-nw85n\") pod \"maas-api-revoke-sub-3a01ea9f-2pcg5\" (UID: \"2efef570-20ac-4b9e-9d4e-f78bc752c0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a01ea9f-2pcg5" Oct 30 21:45:09.586747 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:09.586716 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a01ea9f-2pcg5" Oct 30 21:45:09.983531 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:09.983500 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a01ea9f-2pcg5"] Oct 30 21:45:09.987408 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:45:09.987374 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2efef570_20ac_4b9e_9d4e_f78bc752c0a1.slice/crio-b817ce56c14da0b1257512d2547a7839042bc77e6c36d2ceb93efaf51e2fcccb WatchSource:0}: Error finding container b817ce56c14da0b1257512d2547a7839042bc77e6c36d2ceb93efaf51e2fcccb: Status 404 returned error can't find the container with id b817ce56c14da0b1257512d2547a7839042bc77e6c36d2ceb93efaf51e2fcccb Oct 30 21:45:10.028067 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:10.027995 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a01ea9f-2pcg5" event={"ID":"2efef570-20ac-4b9e-9d4e-f78bc752c0a1","Type":"ContainerStarted","Data":"b817ce56c14da0b1257512d2547a7839042bc77e6c36d2ceb93efaf51e2fcccb"} Oct 30 21:45:10.237598 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:10.237525 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637696f1-pvfq6" Oct 30 21:45:10.274678 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:10.274643 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" Oct 30 21:45:10.275150 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:10.275131 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" Oct 30 21:45:10.276320 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:10.276299 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" Oct 30 21:45:10.363572 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:10.363541 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" Oct 30 21:45:10.364664 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:10.364639 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" Oct 30 21:45:10.365614 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:10.365581 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" Oct 30 21:45:10.404979 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:10.404943 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c-kube-api-access-jmjj6\" (UniqueName: \"kubernetes.io/projected/cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c-kube-api-access-jmjj6\") pod \"cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c\" (UID: \"cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c\") " Oct 30 21:45:10.405142 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:10.405021 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c-maas-api-openshift-ca\") pod \"cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c\" (UID: \"cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c\") " Oct 30 21:45:10.405201 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:10.405160 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c-maas-api-xks-ca\") pod \"cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c\" (UID: \"cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c\") " Oct 30 21:45:10.405429 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:10.405401 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c-maas-api-openshift-ca" pod "cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c" (UID: "cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:10.405997 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:10.405848 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:10.407625 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:10.407599 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c-maas-api-xks-ca" pod "cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c" (UID: "cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:10.407720 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:10.407670 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c-kube-api-access-jmjj6" pod "cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c" (UID: "cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c"). InnerVolumeSpecName "kube-api-access-jmjj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:10.510724 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:10.509626 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:10.510724 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:10.509888 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmjj6\" (UniqueName: \"kubernetes.io/projected/cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c-kube-api-access-jmjj6\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:11.035708 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:11.035679 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637696f1-pvfq6" Oct 30 21:45:11.035708 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:11.035687 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637696f1-pvfq6" event={"ID":"cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c","Type":"ContainerDied","Data":"33ae934b99378317ee985e05f30721ddefdbfddaa9412fc6b8b4c0c126fc2197"} Oct 30 21:45:11.036220 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:11.035733 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33ae934b99378317ee985e05f30721ddefdbfddaa9412fc6b8b4c0c126fc2197" Oct 30 21:45:11.037507 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:11.037474 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a01ea9f-2pcg5" event={"ID":"2efef570-20ac-4b9e-9d4e-f78bc752c0a1","Type":"ContainerStarted","Data":"e5d238fc7af74f2cd0e036b58b77302d9ae8726d90df3f92815826c01b58938d"} Oct 30 21:45:11.080966 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:11.080943 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-751f022a-f2hvf"] Oct 30 21:45:11.084391 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:11.084368 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b05894f-ktjr2"] Oct 30 21:45:11.092232 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:11.092209 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-751f022a-f2hvf"] Oct 30 21:45:11.095125 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:11.095102 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b05894f-ktjr2"] Oct 30 21:45:11.159919 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:11.159859 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a01ea9f-2pcg5" podStartSLOduration=2.15984059 podStartE2EDuration="2.15984059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:11.158577375 +0000 UTC m=+1886.713277151" watchObservedRunningTime="2026-10-30 21:45:11.15984059 +0000 UTC m=+1886.714540366" Oct 30 21:45:12.045400 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:12.045367 4560 generic.go:356] "Generic (PLEG): container finished" podID="2efef570-20ac-4b9e-9d4e-f78bc752c0a1" containerID="e5d238fc7af74f2cd0e036b58b77302d9ae8726d90df3f92815826c01b58938d" exitCode=0 Oct 30 21:45:12.045825 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:12.045460 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a01ea9f-2pcg5" event={"ID":"2efef570-20ac-4b9e-9d4e-f78bc752c0a1","Type":"ContainerDied","Data":"e5d238fc7af74f2cd0e036b58b77302d9ae8726d90df3f92815826c01b58938d"} Oct 30 21:45:12.576181 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:12.576143 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1330052e-81d2-474e-ace8-c98d0f8587c6" path="/var/lib/kubelet/pods/1330052e-81d2-474e-ace8-c98d0f8587c6/volumes" Oct 30 21:45:12.576659 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:12.576641 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f044066-13f0-4162-8aae-74ec3062e041" path="/var/lib/kubelet/pods/8f044066-13f0-4162-8aae-74ec3062e041/volumes" Oct 30 21:45:13.201903 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:13.201877 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a01ea9f-2pcg5" Oct 30 21:45:13.349151 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:13.349061 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2efef570-20ac-4b9e-9d4e-f78bc752c0a1-kube-api-access-nw85n\" (UniqueName: \"kubernetes.io/projected/2efef570-20ac-4b9e-9d4e-f78bc752c0a1-kube-api-access-nw85n\") pod \"2efef570-20ac-4b9e-9d4e-f78bc752c0a1\" (UID: \"2efef570-20ac-4b9e-9d4e-f78bc752c0a1\") " Oct 30 21:45:13.349303 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:13.349239 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2efef570-20ac-4b9e-9d4e-f78bc752c0a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2efef570-20ac-4b9e-9d4e-f78bc752c0a1-maas-api-xks-ca\") pod \"2efef570-20ac-4b9e-9d4e-f78bc752c0a1\" (UID: \"2efef570-20ac-4b9e-9d4e-f78bc752c0a1\") " Oct 30 21:45:13.349402 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:13.349325 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2efef570-20ac-4b9e-9d4e-f78bc752c0a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2efef570-20ac-4b9e-9d4e-f78bc752c0a1-maas-api-openshift-ca\") pod \"2efef570-20ac-4b9e-9d4e-f78bc752c0a1\" (UID: \"2efef570-20ac-4b9e-9d4e-f78bc752c0a1\") " Oct 30 21:45:13.349839 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:13.349807 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2efef570-20ac-4b9e-9d4e-f78bc752c0a1-maas-api-openshift-ca" pod "2efef570-20ac-4b9e-9d4e-f78bc752c0a1" (UID: "2efef570-20ac-4b9e-9d4e-f78bc752c0a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:13.350009 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:13.349987 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2efef570-20ac-4b9e-9d4e-f78bc752c0a1-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:13.351413 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:13.351387 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2efef570-20ac-4b9e-9d4e-f78bc752c0a1-kube-api-access-nw85n" pod "2efef570-20ac-4b9e-9d4e-f78bc752c0a1" (UID: "2efef570-20ac-4b9e-9d4e-f78bc752c0a1"). InnerVolumeSpecName "kube-api-access-nw85n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:13.351784 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:13.351764 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2efef570-20ac-4b9e-9d4e-f78bc752c0a1-maas-api-xks-ca" pod "2efef570-20ac-4b9e-9d4e-f78bc752c0a1" (UID: "2efef570-20ac-4b9e-9d4e-f78bc752c0a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:13.451433 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:13.451403 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw85n\" (UniqueName: \"kubernetes.io/projected/2efef570-20ac-4b9e-9d4e-f78bc752c0a1-kube-api-access-nw85n\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:13.451433 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:13.451433 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2efef570-20ac-4b9e-9d4e-f78bc752c0a1-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:13.774063 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:13.774027 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78afec61-7vcnz"] Oct 30 21:45:13.774603 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:13.774584 4560 cpu_manager.go:414] "containerMap: removing container" podUID="cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c" containerName="revoke-keys" Oct 30 21:45:13.774603 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:13.774603 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c" containerName="revoke-keys" Oct 30 21:45:13.774736 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:13.774619 4560 cpu_manager.go:414] "containerMap: removing container" podUID="2efef570-20ac-4b9e-9d4e-f78bc752c0a1" containerName="revoke-keys" Oct 30 21:45:13.774736 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:13.774624 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2efef570-20ac-4b9e-9d4e-f78bc752c0a1" containerName="revoke-keys" Oct 30 21:45:13.774736 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:13.774732 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c" containerName="revoke-keys" Oct 30 21:45:13.774820 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:13.774740 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="2efef570-20ac-4b9e-9d4e-f78bc752c0a1" containerName="revoke-keys" Oct 30 21:45:13.781024 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:13.780996 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78afec61-7vcnz" Oct 30 21:45:13.833649 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:13.833616 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78afec61-7vcnz"] Oct 30 21:45:13.958261 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:13.958227 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abbc55ac-7b18-4926-a0ba-7217a13bdc41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78afec61-7vcnz\" (UID: \"abbc55ac-7b18-4926-a0ba-7217a13bdc41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78afec61-7vcnz" Oct 30 21:45:13.958440 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:13.958284 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abbc55ac-7b18-4926-a0ba-7217a13bdc41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78afec61-7vcnz\" (UID: \"abbc55ac-7b18-4926-a0ba-7217a13bdc41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78afec61-7vcnz" Oct 30 21:45:13.958440 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:13.958421 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw8p6\" (UniqueName: \"kubernetes.io/projected/abbc55ac-7b18-4926-a0ba-7217a13bdc41-kube-api-access-gw8p6\") pod \"maas-api-revoke-sub-78afec61-7vcnz\" (UID: \"abbc55ac-7b18-4926-a0ba-7217a13bdc41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78afec61-7vcnz" Oct 30 21:45:14.059185 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:14.059107 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw8p6\" (UniqueName: \"kubernetes.io/projected/abbc55ac-7b18-4926-a0ba-7217a13bdc41-kube-api-access-gw8p6\") pod \"maas-api-revoke-sub-78afec61-7vcnz\" (UID: \"abbc55ac-7b18-4926-a0ba-7217a13bdc41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78afec61-7vcnz" Oct 30 21:45:14.059320 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:14.059282 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abbc55ac-7b18-4926-a0ba-7217a13bdc41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78afec61-7vcnz\" (UID: \"abbc55ac-7b18-4926-a0ba-7217a13bdc41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78afec61-7vcnz" Oct 30 21:45:14.059407 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:14.059365 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abbc55ac-7b18-4926-a0ba-7217a13bdc41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78afec61-7vcnz\" (UID: \"abbc55ac-7b18-4926-a0ba-7217a13bdc41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78afec61-7vcnz" Oct 30 21:45:14.059795 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:14.059778 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a01ea9f-2pcg5" Oct 30 21:45:14.059882 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:14.059813 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a01ea9f-2pcg5" event={"ID":"2efef570-20ac-4b9e-9d4e-f78bc752c0a1","Type":"ContainerDied","Data":"b817ce56c14da0b1257512d2547a7839042bc77e6c36d2ceb93efaf51e2fcccb"} Oct 30 21:45:14.059882 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:14.059857 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b817ce56c14da0b1257512d2547a7839042bc77e6c36d2ceb93efaf51e2fcccb" Oct 30 21:45:14.060021 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:14.059992 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abbc55ac-7b18-4926-a0ba-7217a13bdc41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78afec61-7vcnz\" (UID: \"abbc55ac-7b18-4926-a0ba-7217a13bdc41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78afec61-7vcnz" Oct 30 21:45:14.061809 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:14.061790 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abbc55ac-7b18-4926-a0ba-7217a13bdc41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78afec61-7vcnz\" (UID: \"abbc55ac-7b18-4926-a0ba-7217a13bdc41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78afec61-7vcnz" Oct 30 21:45:14.117064 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:14.117038 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw8p6\" (UniqueName: \"kubernetes.io/projected/abbc55ac-7b18-4926-a0ba-7217a13bdc41-kube-api-access-gw8p6\") pod \"maas-api-revoke-sub-78afec61-7vcnz\" (UID: \"abbc55ac-7b18-4926-a0ba-7217a13bdc41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78afec61-7vcnz" Oct 30 21:45:14.395357 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:14.395260 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78afec61-7vcnz" Oct 30 21:45:14.572883 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:14.572842 4560 scope.go:122] "RemoveContainer" containerID="7f93f3d2d251ed0f67f962440be02b170ef3885c94d68b335db470848090b2f3" Oct 30 21:45:14.573066 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:14.573011 4560 scope.go:122] "RemoveContainer" containerID="06c06e1b62c9ed7a5d285f515bd5a5cefeb290c2f213b09b4b08c54c29420ae5" Oct 30 21:45:14.770132 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:14.770081 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78afec61-7vcnz"] Oct 30 21:45:14.773832 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:45:14.773806 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabbc55ac_7b18_4926_a0ba_7217a13bdc41.slice/crio-3e104ef3a83c872c125c7bd25bcf40e7d605f1168ba7eed0c1601f5f2c1917cf WatchSource:0}: Error finding container 3e104ef3a83c872c125c7bd25bcf40e7d605f1168ba7eed0c1601f5f2c1917cf: Status 404 returned error can't find the container with id 3e104ef3a83c872c125c7bd25bcf40e7d605f1168ba7eed0c1601f5f2c1917cf Oct 30 21:45:15.068529 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:15.068499 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889945-6blmw" event={"ID":"0e9ffebf-af98-439e-82b5-7459d1a6538d","Type":"ContainerStarted","Data":"4ff345539ebb0d511539dc5b3221364e1ca718e126c19cc7919cd8f9df815152"} Oct 30 21:45:15.070378 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:15.070356 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a2dced-29889945-w755r" event={"ID":"66765898-e05f-4458-ae55-d69be88bac6a","Type":"ContainerStarted","Data":"cc373fed10fddad321edde9961005365c4a287eedd2bce635b106c7f20e74e1d"} Oct 30 21:45:15.071786 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:15.071767 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78afec61-7vcnz" event={"ID":"abbc55ac-7b18-4926-a0ba-7217a13bdc41","Type":"ContainerStarted","Data":"be46a34984458e2c8b231b9c9b3012acfc8c6e220f3a2815e42d0b235e3a513f"} Oct 30 21:45:15.071875 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:15.071799 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78afec61-7vcnz" event={"ID":"abbc55ac-7b18-4926-a0ba-7217a13bdc41","Type":"ContainerStarted","Data":"3e104ef3a83c872c125c7bd25bcf40e7d605f1168ba7eed0c1601f5f2c1917cf"} Oct 30 21:45:15.327547 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:15.327438 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a2dced-29889945-w755r" podStartSLOduration=15.327422487 podStartE2EDuration="15.327422487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:15.324961158 +0000 UTC m=+1890.879660936" watchObservedRunningTime="2026-10-30 21:45:15.327422487 +0000 UTC m=+1890.882122264" Oct 30 21:45:15.439353 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:15.439298 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78afec61-7vcnz" podStartSLOduration=2.439284072 podStartE2EDuration="2.439284072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:15.435101545 +0000 UTC m=+1890.989801508" watchObservedRunningTime="2026-10-30 21:45:15.439284072 +0000 UTC m=+1890.993983848" Oct 30 21:45:15.569745 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:15.569722 4560 scope.go:122] "RemoveContainer" containerID="42854d8f5b5bcbb431db8f390e2032edb55eb53e6fdfc184f757e451eeac0300" Oct 30 21:45:16.081306 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:16.081275 4560 generic.go:356] "Generic (PLEG): container finished" podID="66765898-e05f-4458-ae55-d69be88bac6a" containerID="cc373fed10fddad321edde9961005365c4a287eedd2bce635b106c7f20e74e1d" exitCode=60 Oct 30 21:45:16.081484 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:16.081374 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a2dced-29889945-w755r" event={"ID":"66765898-e05f-4458-ae55-d69be88bac6a","Type":"ContainerDied","Data":"cc373fed10fddad321edde9961005365c4a287eedd2bce635b106c7f20e74e1d"} Oct 30 21:45:16.081484 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:16.081436 4560 scope.go:122] "RemoveContainer" containerID="06c06e1b62c9ed7a5d285f515bd5a5cefeb290c2f213b09b4b08c54c29420ae5" Oct 30 21:45:16.081826 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:16.081804 4560 scope.go:122] "RemoveContainer" containerID="cc373fed10fddad321edde9961005365c4a287eedd2bce635b106c7f20e74e1d" Oct 30 21:45:16.082124 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:45:16.082100 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-a2dced-29889945-w755r_odh-ai-gateway-infra(66765898-e05f-4458-ae55-d69be88bac6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a2dced-29889945-w755r" podUID="66765898-e05f-4458-ae55-d69be88bac6a" Oct 30 21:45:16.083564 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:16.083545 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ws6mz" event={"ID":"9f301bcc-2a99-4dd3-bddc-b5af224d83f5","Type":"ContainerStarted","Data":"a3a580d643ead7e5506f2b4b05af4bfa63c02a890324a384de0cd6fad566588d"} Oct 30 21:45:16.085058 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:16.085038 4560 generic.go:356] "Generic (PLEG): container finished" podID="abbc55ac-7b18-4926-a0ba-7217a13bdc41" containerID="be46a34984458e2c8b231b9c9b3012acfc8c6e220f3a2815e42d0b235e3a513f" exitCode=0 Oct 30 21:45:16.085145 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:16.085116 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78afec61-7vcnz" event={"ID":"abbc55ac-7b18-4926-a0ba-7217a13bdc41","Type":"ContainerDied","Data":"be46a34984458e2c8b231b9c9b3012acfc8c6e220f3a2815e42d0b235e3a513f"} Oct 30 21:45:16.086916 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:16.086900 4560 generic.go:356] "Generic (PLEG): container finished" podID="0e9ffebf-af98-439e-82b5-7459d1a6538d" containerID="4ff345539ebb0d511539dc5b3221364e1ca718e126c19cc7919cd8f9df815152" exitCode=60 Oct 30 21:45:16.086994 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:16.086940 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889945-6blmw" event={"ID":"0e9ffebf-af98-439e-82b5-7459d1a6538d","Type":"ContainerDied","Data":"4ff345539ebb0d511539dc5b3221364e1ca718e126c19cc7919cd8f9df815152"} Oct 30 21:45:16.087207 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:16.087193 4560 scope.go:122] "RemoveContainer" containerID="4ff345539ebb0d511539dc5b3221364e1ca718e126c19cc7919cd8f9df815152" Oct 30 21:45:16.087424 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:45:16.087404 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-554f46-29889945-6blmw_odh-ai-gateway-infra(0e9ffebf-af98-439e-82b5-7459d1a6538d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889945-6blmw" podUID="0e9ffebf-af98-439e-82b5-7459d1a6538d" Oct 30 21:45:16.094061 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:16.094044 4560 scope.go:122] "RemoveContainer" containerID="7f93f3d2d251ed0f67f962440be02b170ef3885c94d68b335db470848090b2f3" Oct 30 21:45:16.239305 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:16.239273 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889945-6blmw"] Oct 30 21:45:16.338548 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:16.338519 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a2dced-29889945-w755r"] Oct 30 21:45:16.569023 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:16.568998 4560 scope.go:122] "RemoveContainer" containerID="7af3290e705b277167391a217fde730e53daff69165b2068c20c3c256afd275c" Oct 30 21:45:17.099849 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.099820 4560 generic.go:356] "Generic (PLEG): container finished" podID="9f301bcc-2a99-4dd3-bddc-b5af224d83f5" containerID="a3a580d643ead7e5506f2b4b05af4bfa63c02a890324a384de0cd6fad566588d" exitCode=60 Oct 30 21:45:17.099962 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.099891 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ws6mz" event={"ID":"9f301bcc-2a99-4dd3-bddc-b5af224d83f5","Type":"ContainerDied","Data":"a3a580d643ead7e5506f2b4b05af4bfa63c02a890324a384de0cd6fad566588d"} Oct 30 21:45:17.099962 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.099956 4560 scope.go:122] "RemoveContainer" containerID="42854d8f5b5bcbb431db8f390e2032edb55eb53e6fdfc184f757e451eeac0300" Oct 30 21:45:17.100323 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.100296 4560 scope.go:122] "RemoveContainer" containerID="a3a580d643ead7e5506f2b4b05af4bfa63c02a890324a384de0cd6fad566588d" Oct 30 21:45:17.100609 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:45:17.100585 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889945-ws6mz_odh-ai-gateway-infra(9f301bcc-2a99-4dd3-bddc-b5af224d83f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ws6mz" podUID="9f301bcc-2a99-4dd3-bddc-b5af224d83f5" Oct 30 21:45:17.102066 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.102043 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889945-kvfr9" event={"ID":"0b0f43b9-38d3-4e37-a0ce-f987574873b9","Type":"ContainerStarted","Data":"2762fbbf89dedd226b280c56db50084b04658440d4ce0b507ac9a00ccca38ebc"} Oct 30 21:45:17.292020 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.291981 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78afec61-7vcnz" Oct 30 21:45:17.292746 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.292720 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh"] Oct 30 21:45:17.293443 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.293422 4560 cpu_manager.go:414] "containerMap: removing container" podUID="abbc55ac-7b18-4926-a0ba-7217a13bdc41" containerName="revoke-keys" Oct 30 21:45:17.293535 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.293444 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abbc55ac-7b18-4926-a0ba-7217a13bdc41" containerName="revoke-keys" Oct 30 21:45:17.293587 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.293568 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="abbc55ac-7b18-4926-a0ba-7217a13bdc41" containerName="revoke-keys" Oct 30 21:45:17.297214 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.297194 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" Oct 30 21:45:17.343428 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.343404 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a2dced-29889945-w755r" Oct 30 21:45:17.347599 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.347551 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889945-6blmw" Oct 30 21:45:17.397465 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.392946 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh"] Oct 30 21:45:17.398683 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.398655 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abbc55ac-7b18-4926-a0ba-7217a13bdc41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abbc55ac-7b18-4926-a0ba-7217a13bdc41-maas-api-openshift-ca\") pod \"abbc55ac-7b18-4926-a0ba-7217a13bdc41\" (UID: \"abbc55ac-7b18-4926-a0ba-7217a13bdc41\") " Oct 30 21:45:17.398797 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.398722 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abbc55ac-7b18-4926-a0ba-7217a13bdc41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abbc55ac-7b18-4926-a0ba-7217a13bdc41-maas-api-xks-ca\") pod \"abbc55ac-7b18-4926-a0ba-7217a13bdc41\" (UID: \"abbc55ac-7b18-4926-a0ba-7217a13bdc41\") " Oct 30 21:45:17.398959 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.398941 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abbc55ac-7b18-4926-a0ba-7217a13bdc41-kube-api-access-gw8p6\" (UniqueName: \"kubernetes.io/projected/abbc55ac-7b18-4926-a0ba-7217a13bdc41-kube-api-access-gw8p6\") pod \"abbc55ac-7b18-4926-a0ba-7217a13bdc41\" (UID: \"abbc55ac-7b18-4926-a0ba-7217a13bdc41\") " Oct 30 21:45:17.399369 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.399037 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abbc55ac-7b18-4926-a0ba-7217a13bdc41-maas-api-openshift-ca" pod "abbc55ac-7b18-4926-a0ba-7217a13bdc41" (UID: "abbc55ac-7b18-4926-a0ba-7217a13bdc41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:17.399369 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.399201 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b007cff0-0922-4fbf-b4fc-10aaf629a952-workload-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh\" (UID: \"b007cff0-0922-4fbf-b4fc-10aaf629a952\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" Oct 30 21:45:17.399369 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.399268 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b007cff0-0922-4fbf-b4fc-10aaf629a952-istio-data\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh\" (UID: \"b007cff0-0922-4fbf-b4fc-10aaf629a952\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" Oct 30 21:45:17.399369 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.399292 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7pvq\" (UniqueName: \"kubernetes.io/projected/b007cff0-0922-4fbf-b4fc-10aaf629a952-kube-api-access-t7pvq\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh\" (UID: \"b007cff0-0922-4fbf-b4fc-10aaf629a952\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" Oct 30 21:45:17.399613 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.399389 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b007cff0-0922-4fbf-b4fc-10aaf629a952-istio-envoy\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh\" (UID: \"b007cff0-0922-4fbf-b4fc-10aaf629a952\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" Oct 30 21:45:17.399613 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.399473 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b007cff0-0922-4fbf-b4fc-10aaf629a952-workload-certs\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh\" (UID: \"b007cff0-0922-4fbf-b4fc-10aaf629a952\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" Oct 30 21:45:17.399613 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.399516 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b007cff0-0922-4fbf-b4fc-10aaf629a952-istio-podinfo\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh\" (UID: \"b007cff0-0922-4fbf-b4fc-10aaf629a952\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" Oct 30 21:45:17.399765 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.399630 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b007cff0-0922-4fbf-b4fc-10aaf629a952-istio-token\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh\" (UID: \"b007cff0-0922-4fbf-b4fc-10aaf629a952\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" Oct 30 21:45:17.399765 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.399678 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b007cff0-0922-4fbf-b4fc-10aaf629a952-istiod-ca-cert\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh\" (UID: \"b007cff0-0922-4fbf-b4fc-10aaf629a952\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" Oct 30 21:45:17.399765 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.399728 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b007cff0-0922-4fbf-b4fc-10aaf629a952-credential-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh\" (UID: \"b007cff0-0922-4fbf-b4fc-10aaf629a952\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" Oct 30 21:45:17.399904 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.399802 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abbc55ac-7b18-4926-a0ba-7217a13bdc41-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:17.401980 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.401955 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abbc55ac-7b18-4926-a0ba-7217a13bdc41-kube-api-access-gw8p6" pod "abbc55ac-7b18-4926-a0ba-7217a13bdc41" (UID: "abbc55ac-7b18-4926-a0ba-7217a13bdc41"). InnerVolumeSpecName "kube-api-access-gw8p6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:17.402107 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.402089 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abbc55ac-7b18-4926-a0ba-7217a13bdc41-maas-api-xks-ca" pod "abbc55ac-7b18-4926-a0ba-7217a13bdc41" (UID: "abbc55ac-7b18-4926-a0ba-7217a13bdc41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:17.500588 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.500562 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66765898-e05f-4458-ae55-d69be88bac6a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66765898-e05f-4458-ae55-d69be88bac6a-maas-api-service-ca\") pod \"66765898-e05f-4458-ae55-d69be88bac6a\" (UID: \"66765898-e05f-4458-ae55-d69be88bac6a\") " Oct 30 21:45:17.500735 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.500679 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e9ffebf-af98-439e-82b5-7459d1a6538d-kube-api-access-zfk6z\" (UniqueName: \"kubernetes.io/projected/0e9ffebf-af98-439e-82b5-7459d1a6538d-kube-api-access-zfk6z\") pod \"0e9ffebf-af98-439e-82b5-7459d1a6538d\" (UID: \"0e9ffebf-af98-439e-82b5-7459d1a6538d\") " Oct 30 21:45:17.500735 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.500727 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e9ffebf-af98-439e-82b5-7459d1a6538d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e9ffebf-af98-439e-82b5-7459d1a6538d-maas-api-service-ca\") pod \"0e9ffebf-af98-439e-82b5-7459d1a6538d\" (UID: \"0e9ffebf-af98-439e-82b5-7459d1a6538d\") " Oct 30 21:45:17.500988 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.500788 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66765898-e05f-4458-ae55-d69be88bac6a-kube-api-access-qqtwv\" (UniqueName: \"kubernetes.io/projected/66765898-e05f-4458-ae55-d69be88bac6a-kube-api-access-qqtwv\") pod \"66765898-e05f-4458-ae55-d69be88bac6a\" (UID: \"66765898-e05f-4458-ae55-d69be88bac6a\") " Oct 30 21:45:17.501065 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.500993 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66765898-e05f-4458-ae55-d69be88bac6a-maas-api-service-ca" pod "66765898-e05f-4458-ae55-d69be88bac6a" (UID: "66765898-e05f-4458-ae55-d69be88bac6a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:17.501122 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.501059 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b007cff0-0922-4fbf-b4fc-10aaf629a952-workload-certs\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh\" (UID: \"b007cff0-0922-4fbf-b4fc-10aaf629a952\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" Oct 30 21:45:17.501122 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.501113 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b007cff0-0922-4fbf-b4fc-10aaf629a952-istio-podinfo\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh\" (UID: \"b007cff0-0922-4fbf-b4fc-10aaf629a952\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" Oct 30 21:45:17.501237 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.501219 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b007cff0-0922-4fbf-b4fc-10aaf629a952-istio-token\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh\" (UID: \"b007cff0-0922-4fbf-b4fc-10aaf629a952\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" Oct 30 21:45:17.501305 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.501282 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b007cff0-0922-4fbf-b4fc-10aaf629a952-istiod-ca-cert\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh\" (UID: \"b007cff0-0922-4fbf-b4fc-10aaf629a952\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" Oct 30 21:45:17.501382 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.501332 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b007cff0-0922-4fbf-b4fc-10aaf629a952-credential-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh\" (UID: \"b007cff0-0922-4fbf-b4fc-10aaf629a952\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" Oct 30 21:45:17.501443 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.501404 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b007cff0-0922-4fbf-b4fc-10aaf629a952-workload-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh\" (UID: \"b007cff0-0922-4fbf-b4fc-10aaf629a952\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" Oct 30 21:45:17.501498 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.501465 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b007cff0-0922-4fbf-b4fc-10aaf629a952-istio-data\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh\" (UID: \"b007cff0-0922-4fbf-b4fc-10aaf629a952\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" Oct 30 21:45:17.501498 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.501492 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7pvq\" (UniqueName: \"kubernetes.io/projected/b007cff0-0922-4fbf-b4fc-10aaf629a952-kube-api-access-t7pvq\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh\" (UID: \"b007cff0-0922-4fbf-b4fc-10aaf629a952\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" Oct 30 21:45:17.501761 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.501558 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b007cff0-0922-4fbf-b4fc-10aaf629a952-istio-envoy\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh\" (UID: \"b007cff0-0922-4fbf-b4fc-10aaf629a952\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" Oct 30 21:45:17.501761 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.501648 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66765898-e05f-4458-ae55-d69be88bac6a-maas-api-service-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:17.501761 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.501665 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw8p6\" (UniqueName: \"kubernetes.io/projected/abbc55ac-7b18-4926-a0ba-7217a13bdc41-kube-api-access-gw8p6\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:17.501761 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.501681 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abbc55ac-7b18-4926-a0ba-7217a13bdc41-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:17.502009 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.501986 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b007cff0-0922-4fbf-b4fc-10aaf629a952-workload-certs\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh\" (UID: \"b007cff0-0922-4fbf-b4fc-10aaf629a952\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" Oct 30 21:45:17.503925 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.503904 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b007cff0-0922-4fbf-b4fc-10aaf629a952-istio-envoy\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh\" (UID: \"b007cff0-0922-4fbf-b4fc-10aaf629a952\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" Oct 30 21:45:17.505552 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.504721 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b007cff0-0922-4fbf-b4fc-10aaf629a952-istio-podinfo\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh\" (UID: \"b007cff0-0922-4fbf-b4fc-10aaf629a952\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" Oct 30 21:45:17.505552 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.504778 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b007cff0-0922-4fbf-b4fc-10aaf629a952-workload-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh\" (UID: \"b007cff0-0922-4fbf-b4fc-10aaf629a952\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" Oct 30 21:45:17.505552 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.504807 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b007cff0-0922-4fbf-b4fc-10aaf629a952-istiod-ca-cert\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh\" (UID: \"b007cff0-0922-4fbf-b4fc-10aaf629a952\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" Oct 30 21:45:17.505552 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.501331 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e9ffebf-af98-439e-82b5-7459d1a6538d-maas-api-service-ca" pod "0e9ffebf-af98-439e-82b5-7459d1a6538d" (UID: "0e9ffebf-af98-439e-82b5-7459d1a6538d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:17.505552 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.505037 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b007cff0-0922-4fbf-b4fc-10aaf629a952-credential-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh\" (UID: \"b007cff0-0922-4fbf-b4fc-10aaf629a952\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" Oct 30 21:45:17.505552 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.505020 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e9ffebf-af98-439e-82b5-7459d1a6538d-kube-api-access-zfk6z" pod "0e9ffebf-af98-439e-82b5-7459d1a6538d" (UID: "0e9ffebf-af98-439e-82b5-7459d1a6538d"). InnerVolumeSpecName "kube-api-access-zfk6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:17.505552 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.505057 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b007cff0-0922-4fbf-b4fc-10aaf629a952-istio-data\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh\" (UID: \"b007cff0-0922-4fbf-b4fc-10aaf629a952\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" Oct 30 21:45:17.505552 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.505528 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66765898-e05f-4458-ae55-d69be88bac6a-kube-api-access-qqtwv" pod "66765898-e05f-4458-ae55-d69be88bac6a" (UID: "66765898-e05f-4458-ae55-d69be88bac6a"). InnerVolumeSpecName "kube-api-access-qqtwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:17.519146 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.519122 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75ee6b3c-fxrpq"] Oct 30 21:45:17.519617 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.519600 4560 cpu_manager.go:414] "containerMap: removing container" podUID="66765898-e05f-4458-ae55-d69be88bac6a" containerName="cleanup" Oct 30 21:45:17.519617 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.519615 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66765898-e05f-4458-ae55-d69be88bac6a" containerName="cleanup" Oct 30 21:45:17.519695 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.519625 4560 cpu_manager.go:414] "containerMap: removing container" podUID="0e9ffebf-af98-439e-82b5-7459d1a6538d" containerName="cleanup" Oct 30 21:45:17.519695 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.519631 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e9ffebf-af98-439e-82b5-7459d1a6538d" containerName="cleanup" Oct 30 21:45:17.519695 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.519640 4560 cpu_manager.go:414] "containerMap: removing container" podUID="0e9ffebf-af98-439e-82b5-7459d1a6538d" containerName="cleanup" Oct 30 21:45:17.519695 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.519645 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e9ffebf-af98-439e-82b5-7459d1a6538d" containerName="cleanup" Oct 30 21:45:17.519695 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.519653 4560 cpu_manager.go:414] "containerMap: removing container" podUID="0e9ffebf-af98-439e-82b5-7459d1a6538d" containerName="cleanup" Oct 30 21:45:17.519695 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.519657 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e9ffebf-af98-439e-82b5-7459d1a6538d" containerName="cleanup" Oct 30 21:45:17.519695 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.519680 4560 cpu_manager.go:414] "containerMap: removing container" podUID="66765898-e05f-4458-ae55-d69be88bac6a" containerName="cleanup" Oct 30 21:45:17.519695 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.519684 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66765898-e05f-4458-ae55-d69be88bac6a" containerName="cleanup" Oct 30 21:45:17.519912 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.519769 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="66765898-e05f-4458-ae55-d69be88bac6a" containerName="cleanup" Oct 30 21:45:17.519912 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.519777 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e9ffebf-af98-439e-82b5-7459d1a6538d" containerName="cleanup" Oct 30 21:45:17.519912 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.519783 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e9ffebf-af98-439e-82b5-7459d1a6538d" containerName="cleanup" Oct 30 21:45:17.519912 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.519789 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e9ffebf-af98-439e-82b5-7459d1a6538d" containerName="cleanup" Oct 30 21:45:17.519912 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.519801 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="66765898-e05f-4458-ae55-d69be88bac6a" containerName="cleanup" Oct 30 21:45:17.524110 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.524094 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ee6b3c-fxrpq" Oct 30 21:45:17.553544 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.553515 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ws6mz"] Oct 30 21:45:17.558976 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.558944 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b007cff0-0922-4fbf-b4fc-10aaf629a952-istio-token\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh\" (UID: \"b007cff0-0922-4fbf-b4fc-10aaf629a952\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" Oct 30 21:45:17.562523 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.562497 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7pvq\" (UniqueName: \"kubernetes.io/projected/b007cff0-0922-4fbf-b4fc-10aaf629a952-kube-api-access-t7pvq\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh\" (UID: \"b007cff0-0922-4fbf-b4fc-10aaf629a952\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" Oct 30 21:45:17.584973 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.584950 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75ee6b3c-fxrpq"] Oct 30 21:45:17.603388 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.603278 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9efba11d-653c-47de-9eaf-dc250f12d18c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75ee6b3c-fxrpq\" (UID: \"9efba11d-653c-47de-9eaf-dc250f12d18c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ee6b3c-fxrpq" Oct 30 21:45:17.603388 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.603384 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9efba11d-653c-47de-9eaf-dc250f12d18c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75ee6b3c-fxrpq\" (UID: \"9efba11d-653c-47de-9eaf-dc250f12d18c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ee6b3c-fxrpq" Oct 30 21:45:17.603579 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.603475 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnxjx\" (UniqueName: \"kubernetes.io/projected/9efba11d-653c-47de-9eaf-dc250f12d18c-kube-api-access-qnxjx\") pod \"maas-api-revoke-sub-75ee6b3c-fxrpq\" (UID: \"9efba11d-653c-47de-9eaf-dc250f12d18c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ee6b3c-fxrpq" Oct 30 21:45:17.603579 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.603568 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqtwv\" (UniqueName: \"kubernetes.io/projected/66765898-e05f-4458-ae55-d69be88bac6a-kube-api-access-qqtwv\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:17.603680 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.603591 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfk6z\" (UniqueName: \"kubernetes.io/projected/0e9ffebf-af98-439e-82b5-7459d1a6538d-kube-api-access-zfk6z\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:17.603680 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.603606 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e9ffebf-af98-439e-82b5-7459d1a6538d-maas-api-service-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:17.614883 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.614853 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" Oct 30 21:45:17.673226 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.673194 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb"] Oct 30 21:45:17.673720 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.673706 4560 cpu_manager.go:414] "containerMap: removing container" podUID="66765898-e05f-4458-ae55-d69be88bac6a" containerName="cleanup" Oct 30 21:45:17.673785 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.673721 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66765898-e05f-4458-ae55-d69be88bac6a" containerName="cleanup" Oct 30 21:45:17.673834 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.673819 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="66765898-e05f-4458-ae55-d69be88bac6a" containerName="cleanup" Oct 30 21:45:17.678308 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.678289 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" Oct 30 21:45:17.706805 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.705145 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9efba11d-653c-47de-9eaf-dc250f12d18c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75ee6b3c-fxrpq\" (UID: \"9efba11d-653c-47de-9eaf-dc250f12d18c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ee6b3c-fxrpq" Oct 30 21:45:17.706805 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.705196 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnxjx\" (UniqueName: \"kubernetes.io/projected/9efba11d-653c-47de-9eaf-dc250f12d18c-kube-api-access-qnxjx\") pod \"maas-api-revoke-sub-75ee6b3c-fxrpq\" (UID: \"9efba11d-653c-47de-9eaf-dc250f12d18c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ee6b3c-fxrpq" Oct 30 21:45:17.706805 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.705468 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9efba11d-653c-47de-9eaf-dc250f12d18c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75ee6b3c-fxrpq\" (UID: \"9efba11d-653c-47de-9eaf-dc250f12d18c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ee6b3c-fxrpq" Oct 30 21:45:17.706805 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.706241 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9efba11d-653c-47de-9eaf-dc250f12d18c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75ee6b3c-fxrpq\" (UID: \"9efba11d-653c-47de-9eaf-dc250f12d18c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ee6b3c-fxrpq" Oct 30 21:45:17.709662 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.709641 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9efba11d-653c-47de-9eaf-dc250f12d18c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75ee6b3c-fxrpq\" (UID: \"9efba11d-653c-47de-9eaf-dc250f12d18c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ee6b3c-fxrpq" Oct 30 21:45:17.758195 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.758160 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb"] Oct 30 21:45:17.764050 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.764009 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnxjx\" (UniqueName: \"kubernetes.io/projected/9efba11d-653c-47de-9eaf-dc250f12d18c-kube-api-access-qnxjx\") pod \"maas-api-revoke-sub-75ee6b3c-fxrpq\" (UID: \"9efba11d-653c-47de-9eaf-dc250f12d18c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ee6b3c-fxrpq" Oct 30 21:45:17.806736 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.806698 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-workload-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb\" (UID: \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" Oct 30 21:45:17.806870 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.806738 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-istio-data\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb\" (UID: \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" Oct 30 21:45:17.806936 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.806917 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-istio-podinfo\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb\" (UID: \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" Oct 30 21:45:17.806970 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.806944 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-istio-token\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb\" (UID: \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" Oct 30 21:45:17.807049 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.807028 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-workload-certs\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb\" (UID: \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" Oct 30 21:45:17.807128 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.807105 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-istio-envoy\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb\" (UID: \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" Oct 30 21:45:17.807185 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.807171 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-istiod-ca-cert\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb\" (UID: \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" Oct 30 21:45:17.807256 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.807240 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-credential-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb\" (UID: \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" Oct 30 21:45:17.807399 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.807380 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nhfh\" (UniqueName: \"kubernetes.io/projected/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-kube-api-access-2nhfh\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb\" (UID: \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" Oct 30 21:45:17.836941 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.836914 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ee6b3c-fxrpq" Oct 30 21:45:17.908679 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.908651 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nhfh\" (UniqueName: \"kubernetes.io/projected/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-kube-api-access-2nhfh\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb\" (UID: \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" Oct 30 21:45:17.908947 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.908926 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-workload-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb\" (UID: \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" Oct 30 21:45:17.909495 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.909471 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-istio-data\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb\" (UID: \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" Oct 30 21:45:17.909726 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.909708 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-istio-podinfo\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb\" (UID: \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" Oct 30 21:45:17.909862 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.909845 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-istio-token\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb\" (UID: \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" Oct 30 21:45:17.910057 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.910039 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-workload-certs\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb\" (UID: \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" Oct 30 21:45:17.910261 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.910244 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-istio-envoy\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb\" (UID: \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" Oct 30 21:45:17.910448 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.910420 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-istiod-ca-cert\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb\" (UID: \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" Oct 30 21:45:17.910603 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.910575 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-workload-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb\" (UID: \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" Oct 30 21:45:17.910710 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.910623 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-credential-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb\" (UID: \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" Oct 30 21:45:17.910811 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.910793 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-workload-certs\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb\" (UID: \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" Oct 30 21:45:17.911092 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.911065 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-credential-socket\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb\" (UID: \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" Oct 30 21:45:17.912010 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.910317 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-istio-data\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb\" (UID: \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" Oct 30 21:45:17.912010 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.911962 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-istiod-ca-cert\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb\" (UID: \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" Oct 30 21:45:17.913718 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.913696 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-istio-envoy\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb\" (UID: \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" Oct 30 21:45:17.913856 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.913833 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-istio-podinfo\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb\" (UID: \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" Oct 30 21:45:17.967725 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.967693 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nhfh\" (UniqueName: \"kubernetes.io/projected/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-kube-api-access-2nhfh\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb\" (UID: \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" Oct 30 21:45:17.968806 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:17.968784 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-istio-token\") pod \"e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb\" (UID: \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\") " pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" Oct 30 21:45:18.011729 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.011702 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" Oct 30 21:45:18.086239 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.086212 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh"] Oct 30 21:45:18.089701 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:45:18.089670 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb007cff0_0922_4fbf_b4fc_10aaf629a952.slice/crio-6a6808889dfa988a43ba69c15db6064311e77efbe69134d6105667c1ff933400 WatchSource:0}: Error finding container 6a6808889dfa988a43ba69c15db6064311e77efbe69134d6105667c1ff933400: Status 404 returned error can't find the container with id 6a6808889dfa988a43ba69c15db6064311e77efbe69134d6105667c1ff933400 Oct 30 21:45:18.092488 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.092454 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:45:18.092575 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.092533 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:45:18.092631 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.092582 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:45:18.113979 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.113953 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889945-6blmw" event={"ID":"0e9ffebf-af98-439e-82b5-7459d1a6538d","Type":"ContainerDied","Data":"760bb281c3368f5e00485224ab74348304418ae98e4fb1e8f4645d6145acfed5"} Oct 30 21:45:18.114091 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.114018 4560 scope.go:122] "RemoveContainer" containerID="4ff345539ebb0d511539dc5b3221364e1ca718e126c19cc7919cd8f9df815152" Oct 30 21:45:18.114136 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.114096 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889945-6blmw" Oct 30 21:45:18.115989 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.115965 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a2dced-29889945-w755r" Oct 30 21:45:18.116103 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.116056 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a2dced-29889945-w755r" event={"ID":"66765898-e05f-4458-ae55-d69be88bac6a","Type":"ContainerDied","Data":"b7ea6b010ae7c8a262ce513a42d0ce278a287f7413902bb9420f1b9981192cbc"} Oct 30 21:45:18.125657 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.125512 4560 generic.go:356] "Generic (PLEG): container finished" podID="0b0f43b9-38d3-4e37-a0ce-f987574873b9" containerID="2762fbbf89dedd226b280c56db50084b04658440d4ce0b507ac9a00ccca38ebc" exitCode=60 Oct 30 21:45:18.125657 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.125587 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889945-kvfr9" event={"ID":"0b0f43b9-38d3-4e37-a0ce-f987574873b9","Type":"ContainerDied","Data":"2762fbbf89dedd226b280c56db50084b04658440d4ce0b507ac9a00ccca38ebc"} Oct 30 21:45:18.126021 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.126000 4560 scope.go:122] "RemoveContainer" containerID="2762fbbf89dedd226b280c56db50084b04658440d4ce0b507ac9a00ccca38ebc" Oct 30 21:45:18.126309 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:45:18.126288 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-71aab5-29889945-kvfr9_odh-ai-gateway-infra(0b0f43b9-38d3-4e37-a0ce-f987574873b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889945-kvfr9" podUID="0b0f43b9-38d3-4e37-a0ce-f987574873b9" Oct 30 21:45:18.127178 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.127150 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" event={"ID":"b007cff0-0922-4fbf-b4fc-10aaf629a952","Type":"ContainerStarted","Data":"6a6808889dfa988a43ba69c15db6064311e77efbe69134d6105667c1ff933400"} Oct 30 21:45:18.136850 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.136818 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78afec61-7vcnz" event={"ID":"abbc55ac-7b18-4926-a0ba-7217a13bdc41","Type":"ContainerDied","Data":"3e104ef3a83c872c125c7bd25bcf40e7d605f1168ba7eed0c1601f5f2c1917cf"} Oct 30 21:45:18.136971 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.136919 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e104ef3a83c872c125c7bd25bcf40e7d605f1168ba7eed0c1601f5f2c1917cf" Oct 30 21:45:18.137038 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.137013 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78afec61-7vcnz" Oct 30 21:45:18.144425 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.144400 4560 scope.go:122] "RemoveContainer" containerID="cc373fed10fddad321edde9961005365c4a287eedd2bce635b106c7f20e74e1d" Oct 30 21:45:18.161373 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.161333 4560 scope.go:122] "RemoveContainer" containerID="7af3290e705b277167391a217fde730e53daff69165b2068c20c3c256afd275c" Oct 30 21:45:18.312127 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.312093 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a2dced-29889945-w755r"] Oct 30 21:45:18.316280 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.316251 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a2dced-29889945-w755r"] Oct 30 21:45:18.323957 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.323936 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ws6mz" Oct 30 21:45:18.347703 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.347649 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75ee6b3c-fxrpq"] Oct 30 21:45:18.350814 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:45:18.350486 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9efba11d_653c_47de_9eaf_dc250f12d18c.slice/crio-a3706c99a8391adb5a8a9e51ee1087ea73435bbefa25c0de2c432a447ac295ba WatchSource:0}: Error finding container a3706c99a8391adb5a8a9e51ee1087ea73435bbefa25c0de2c432a447ac295ba: Status 404 returned error can't find the container with id a3706c99a8391adb5a8a9e51ee1087ea73435bbefa25c0de2c432a447ac295ba Oct 30 21:45:18.404606 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.404579 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb"] Oct 30 21:45:18.408797 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:45:18.408772 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaf8da80_7e8e_40a3_8c88_33b2325ef2aa.slice/crio-ac8118fc948ca453e4eedc572bb5a7dd8e39d57ee143d276a657fcee052adddd WatchSource:0}: Error finding container ac8118fc948ca453e4eedc572bb5a7dd8e39d57ee143d276a657fcee052adddd: Status 404 returned error can't find the container with id ac8118fc948ca453e4eedc572bb5a7dd8e39d57ee143d276a657fcee052adddd Oct 30 21:45:18.411290 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.411256 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:45:18.411394 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.411367 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:45:18.411453 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.411419 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:45:18.422008 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.421985 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889945-6blmw"] Oct 30 21:45:18.424936 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.424913 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f301bcc-2a99-4dd3-bddc-b5af224d83f5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f301bcc-2a99-4dd3-bddc-b5af224d83f5-maas-api-service-ca\") pod \"9f301bcc-2a99-4dd3-bddc-b5af224d83f5\" (UID: \"9f301bcc-2a99-4dd3-bddc-b5af224d83f5\") " Oct 30 21:45:18.425089 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.425075 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f301bcc-2a99-4dd3-bddc-b5af224d83f5-kube-api-access-7j82g\" (UniqueName: \"kubernetes.io/projected/9f301bcc-2a99-4dd3-bddc-b5af224d83f5-kube-api-access-7j82g\") pod \"9f301bcc-2a99-4dd3-bddc-b5af224d83f5\" (UID: \"9f301bcc-2a99-4dd3-bddc-b5af224d83f5\") " Oct 30 21:45:18.425762 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.425736 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f301bcc-2a99-4dd3-bddc-b5af224d83f5-maas-api-service-ca" pod "9f301bcc-2a99-4dd3-bddc-b5af224d83f5" (UID: "9f301bcc-2a99-4dd3-bddc-b5af224d83f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:18.428292 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.428199 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f301bcc-2a99-4dd3-bddc-b5af224d83f5-kube-api-access-7j82g" pod "9f301bcc-2a99-4dd3-bddc-b5af224d83f5" (UID: "9f301bcc-2a99-4dd3-bddc-b5af224d83f5"). InnerVolumeSpecName "kube-api-access-7j82g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:18.428419 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.428378 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889945-6blmw"] Oct 30 21:45:18.510675 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.510647 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx"] Oct 30 21:45:18.511295 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.511277 4560 cpu_manager.go:414] "containerMap: removing container" podUID="9f301bcc-2a99-4dd3-bddc-b5af224d83f5" containerName="cleanup" Oct 30 21:45:18.511430 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.511297 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f301bcc-2a99-4dd3-bddc-b5af224d83f5" containerName="cleanup" Oct 30 21:45:18.511430 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.511310 4560 cpu_manager.go:414] "containerMap: removing container" podUID="9f301bcc-2a99-4dd3-bddc-b5af224d83f5" containerName="cleanup" Oct 30 21:45:18.511430 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.511317 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f301bcc-2a99-4dd3-bddc-b5af224d83f5" containerName="cleanup" Oct 30 21:45:18.511430 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.511362 4560 cpu_manager.go:414] "containerMap: removing container" podUID="9f301bcc-2a99-4dd3-bddc-b5af224d83f5" containerName="cleanup" Oct 30 21:45:18.511430 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.511371 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f301bcc-2a99-4dd3-bddc-b5af224d83f5" containerName="cleanup" Oct 30 21:45:18.512124 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.511504 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f301bcc-2a99-4dd3-bddc-b5af224d83f5" containerName="cleanup" Oct 30 21:45:18.512124 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.511514 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f301bcc-2a99-4dd3-bddc-b5af224d83f5" containerName="cleanup" Oct 30 21:45:18.512124 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.511525 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f301bcc-2a99-4dd3-bddc-b5af224d83f5" containerName="cleanup" Oct 30 21:45:18.518313 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.516438 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" Oct 30 21:45:18.526890 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.526864 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j82g\" (UniqueName: \"kubernetes.io/projected/9f301bcc-2a99-4dd3-bddc-b5af224d83f5-kube-api-access-7j82g\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:18.526890 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.526895 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f301bcc-2a99-4dd3-bddc-b5af224d83f5-maas-api-service-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:18.592890 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.591801 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e9ffebf-af98-439e-82b5-7459d1a6538d" path="/var/lib/kubelet/pods/0e9ffebf-af98-439e-82b5-7459d1a6538d/volumes" Oct 30 21:45:18.592890 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.592481 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66765898-e05f-4458-ae55-d69be88bac6a" path="/var/lib/kubelet/pods/66765898-e05f-4458-ae55-d69be88bac6a/volumes" Oct 30 21:45:18.599378 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.596559 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx"] Oct 30 21:45:18.614775 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.614677 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889945-kvfr9"] Oct 30 21:45:18.639409 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.639319 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb9a4d9-6480-470b-967f-96b99f9ff606-workload-certs\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx\" (UID: \"2bb9a4d9-6480-470b-967f-96b99f9ff606\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" Oct 30 21:45:18.639621 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.639604 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bb9a4d9-6480-470b-967f-96b99f9ff606-istio-token\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx\" (UID: \"2bb9a4d9-6480-470b-967f-96b99f9ff606\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" Oct 30 21:45:18.639735 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.639724 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb9a4d9-6480-470b-967f-96b99f9ff606-workload-socket\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx\" (UID: \"2bb9a4d9-6480-470b-967f-96b99f9ff606\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" Oct 30 21:45:18.639825 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.639815 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb9a4d9-6480-470b-967f-96b99f9ff606-credential-socket\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx\" (UID: \"2bb9a4d9-6480-470b-967f-96b99f9ff606\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" Oct 30 21:45:18.639936 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.639927 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb9a4d9-6480-470b-967f-96b99f9ff606-istio-data\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx\" (UID: \"2bb9a4d9-6480-470b-967f-96b99f9ff606\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" Oct 30 21:45:18.640022 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.640013 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb9a4d9-6480-470b-967f-96b99f9ff606-istio-podinfo\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx\" (UID: \"2bb9a4d9-6480-470b-967f-96b99f9ff606\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" Oct 30 21:45:18.640132 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.640123 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb9a4d9-6480-470b-967f-96b99f9ff606-istiod-ca-cert\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx\" (UID: \"2bb9a4d9-6480-470b-967f-96b99f9ff606\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" Oct 30 21:45:18.640233 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.640223 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8fdk\" (UniqueName: \"kubernetes.io/projected/2bb9a4d9-6480-470b-967f-96b99f9ff606-kube-api-access-s8fdk\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx\" (UID: \"2bb9a4d9-6480-470b-967f-96b99f9ff606\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" Oct 30 21:45:18.640390 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.640376 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb9a4d9-6480-470b-967f-96b99f9ff606-istio-envoy\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx\" (UID: \"2bb9a4d9-6480-470b-967f-96b99f9ff606\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" Oct 30 21:45:18.741649 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.741621 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8fdk\" (UniqueName: \"kubernetes.io/projected/2bb9a4d9-6480-470b-967f-96b99f9ff606-kube-api-access-s8fdk\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx\" (UID: \"2bb9a4d9-6480-470b-967f-96b99f9ff606\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" Oct 30 21:45:18.741806 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.741782 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb9a4d9-6480-470b-967f-96b99f9ff606-istio-envoy\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx\" (UID: \"2bb9a4d9-6480-470b-967f-96b99f9ff606\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" Oct 30 21:45:18.742123 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.742088 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb9a4d9-6480-470b-967f-96b99f9ff606-workload-certs\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx\" (UID: \"2bb9a4d9-6480-470b-967f-96b99f9ff606\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" Oct 30 21:45:18.742496 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.742466 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb9a4d9-6480-470b-967f-96b99f9ff606-workload-certs\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx\" (UID: \"2bb9a4d9-6480-470b-967f-96b99f9ff606\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" Oct 30 21:45:18.742653 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.742556 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bb9a4d9-6480-470b-967f-96b99f9ff606-istio-token\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx\" (UID: \"2bb9a4d9-6480-470b-967f-96b99f9ff606\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" Oct 30 21:45:18.742653 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.742601 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb9a4d9-6480-470b-967f-96b99f9ff606-workload-socket\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx\" (UID: \"2bb9a4d9-6480-470b-967f-96b99f9ff606\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" Oct 30 21:45:18.742889 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.742831 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb9a4d9-6480-470b-967f-96b99f9ff606-credential-socket\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx\" (UID: \"2bb9a4d9-6480-470b-967f-96b99f9ff606\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" Oct 30 21:45:18.742997 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.742949 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb9a4d9-6480-470b-967f-96b99f9ff606-istio-data\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx\" (UID: \"2bb9a4d9-6480-470b-967f-96b99f9ff606\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" Oct 30 21:45:18.742997 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.742978 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb9a4d9-6480-470b-967f-96b99f9ff606-istio-podinfo\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx\" (UID: \"2bb9a4d9-6480-470b-967f-96b99f9ff606\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" Oct 30 21:45:18.743120 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.743017 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb9a4d9-6480-470b-967f-96b99f9ff606-istiod-ca-cert\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx\" (UID: \"2bb9a4d9-6480-470b-967f-96b99f9ff606\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" Oct 30 21:45:18.743516 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.743475 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb9a4d9-6480-470b-967f-96b99f9ff606-credential-socket\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx\" (UID: \"2bb9a4d9-6480-470b-967f-96b99f9ff606\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" Oct 30 21:45:18.743637 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.743508 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb9a4d9-6480-470b-967f-96b99f9ff606-istio-data\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx\" (UID: \"2bb9a4d9-6480-470b-967f-96b99f9ff606\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" Oct 30 21:45:18.743720 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.743700 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb9a4d9-6480-470b-967f-96b99f9ff606-workload-socket\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx\" (UID: \"2bb9a4d9-6480-470b-967f-96b99f9ff606\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" Oct 30 21:45:18.743786 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.743754 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb9a4d9-6480-470b-967f-96b99f9ff606-istiod-ca-cert\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx\" (UID: \"2bb9a4d9-6480-470b-967f-96b99f9ff606\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" Oct 30 21:45:18.744932 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.744908 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb9a4d9-6480-470b-967f-96b99f9ff606-istio-envoy\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx\" (UID: \"2bb9a4d9-6480-470b-967f-96b99f9ff606\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" Oct 30 21:45:18.745807 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.745784 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb9a4d9-6480-470b-967f-96b99f9ff606-istio-podinfo\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx\" (UID: \"2bb9a4d9-6480-470b-967f-96b99f9ff606\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" Oct 30 21:45:18.800923 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.800882 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bb9a4d9-6480-470b-967f-96b99f9ff606-istio-token\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx\" (UID: \"2bb9a4d9-6480-470b-967f-96b99f9ff606\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" Oct 30 21:45:18.801074 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.800981 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8fdk\" (UniqueName: \"kubernetes.io/projected/2bb9a4d9-6480-470b-967f-96b99f9ff606-kube-api-access-s8fdk\") pod \"e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx\" (UID: \"2bb9a4d9-6480-470b-967f-96b99f9ff606\") " pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" Oct 30 21:45:18.841544 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:18.841513 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" Oct 30 21:45:19.186514 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:19.186422 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" event={"ID":"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa","Type":"ContainerStarted","Data":"dc97903049a2d73dc0d3d2931402c73118f0019cf5f6396076ebeedd6f969433"} Oct 30 21:45:19.186514 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:19.186484 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" event={"ID":"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa","Type":"ContainerStarted","Data":"ac8118fc948ca453e4eedc572bb5a7dd8e39d57ee143d276a657fcee052adddd"} Oct 30 21:45:19.188950 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:19.188916 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ws6mz" Oct 30 21:45:19.189060 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:19.188918 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ws6mz" event={"ID":"9f301bcc-2a99-4dd3-bddc-b5af224d83f5","Type":"ContainerDied","Data":"613facecced298cb098733e4242a91042ce5d4a3f8b31e2ea957d295e6e9653b"} Oct 30 21:45:19.189060 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:19.189056 4560 scope.go:122] "RemoveContainer" containerID="a3a580d643ead7e5506f2b4b05af4bfa63c02a890324a384de0cd6fad566588d" Oct 30 21:45:19.193063 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:19.193038 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ee6b3c-fxrpq" event={"ID":"9efba11d-653c-47de-9eaf-dc250f12d18c","Type":"ContainerStarted","Data":"32a9401f562311c7b2adff2eed2206f78d3f869632910a1a1af42aa0f1da9ce5"} Oct 30 21:45:19.193146 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:19.193085 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ee6b3c-fxrpq" event={"ID":"9efba11d-653c-47de-9eaf-dc250f12d18c","Type":"ContainerStarted","Data":"a3706c99a8391adb5a8a9e51ee1087ea73435bbefa25c0de2c432a447ac295ba"} Oct 30 21:45:19.195368 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:19.195326 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" event={"ID":"b007cff0-0922-4fbf-b4fc-10aaf629a952","Type":"ContainerStarted","Data":"84fb0557cf8858144316819eed9efcf1a3b0c8f296164895a19cba8e2f427acc"} Oct 30 21:45:19.262361 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:19.262252 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx"] Oct 30 21:45:19.286962 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:19.286920 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:45:19.287131 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:19.287014 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:45:19.287131 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:19.287069 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:45:19.393425 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:19.390300 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" podStartSLOduration=2.390280627 podStartE2EDuration="2.390280627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:19.358844251 +0000 UTC m=+1894.913544029" watchObservedRunningTime="2026-10-30 21:45:19.390280627 +0000 UTC m=+1894.944980405" Oct 30 21:45:19.501740 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:19.501709 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889945-kvfr9" Oct 30 21:45:19.652454 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:19.615292 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" Oct 30 21:45:19.652454 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:19.651223 4560 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:45:19.652454 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:19.651297 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" podUID="b007cff0-0922-4fbf-b4fc-10aaf629a952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 21:45:19.709101 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:19.707117 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b0f43b9-38d3-4e37-a0ce-f987574873b9-kube-api-access-htrml\" (UniqueName: \"kubernetes.io/projected/0b0f43b9-38d3-4e37-a0ce-f987574873b9-kube-api-access-htrml\") pod \"0b0f43b9-38d3-4e37-a0ce-f987574873b9\" (UID: \"0b0f43b9-38d3-4e37-a0ce-f987574873b9\") " Oct 30 21:45:19.709101 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:19.707210 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b0f43b9-38d3-4e37-a0ce-f987574873b9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b0f43b9-38d3-4e37-a0ce-f987574873b9-maas-api-service-ca\") pod \"0b0f43b9-38d3-4e37-a0ce-f987574873b9\" (UID: \"0b0f43b9-38d3-4e37-a0ce-f987574873b9\") " Oct 30 21:45:19.709101 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:19.709031 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b0f43b9-38d3-4e37-a0ce-f987574873b9-maas-api-service-ca" pod "0b0f43b9-38d3-4e37-a0ce-f987574873b9" (UID: "0b0f43b9-38d3-4e37-a0ce-f987574873b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:19.723515 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:19.723470 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b0f43b9-38d3-4e37-a0ce-f987574873b9-kube-api-access-htrml" pod "0b0f43b9-38d3-4e37-a0ce-f987574873b9" (UID: "0b0f43b9-38d3-4e37-a0ce-f987574873b9"). InnerVolumeSpecName "kube-api-access-htrml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:19.730362 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:45:19.726771 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:45:19.789028 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:19.788905 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" podStartSLOduration=2.7888832199999998 podStartE2EDuration="2.78888322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:19.633385817 +0000 UTC m=+1895.188085595" watchObservedRunningTime="2026-10-30 21:45:19.78888322 +0000 UTC m=+1895.343582998" Oct 30 21:45:19.793364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:19.791796 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ws6mz"] Oct 30 21:45:19.798618 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:19.798594 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-ws6mz"] Oct 30 21:45:19.809269 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:19.809233 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htrml\" (UniqueName: \"kubernetes.io/projected/0b0f43b9-38d3-4e37-a0ce-f987574873b9-kube-api-access-htrml\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:19.809431 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:19.809273 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b0f43b9-38d3-4e37-a0ce-f987574873b9-maas-api-service-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:19.886626 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:19.886560 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ee6b3c-fxrpq" podStartSLOduration=2.886545876 podStartE2EDuration="2.886545876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:19.877794575 +0000 UTC m=+1895.432494351" watchObservedRunningTime="2026-10-30 21:45:19.886545876 +0000 UTC m=+1895.441245653" Oct 30 21:45:20.012299 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:20.012223 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" Oct 30 21:45:20.014226 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:20.014198 4560 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:45:20.014385 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:20.014259 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" podUID="aaf8da80-7e8e-40a3-8c88-33b2325ef2aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 21:45:20.214368 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:20.214279 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889945-kvfr9" event={"ID":"0b0f43b9-38d3-4e37-a0ce-f987574873b9","Type":"ContainerDied","Data":"34df49300348922e6929ae7b052f2fec299f2bcac7ba6c51d5a7ccfc50117ec6"} Oct 30 21:45:20.214555 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:20.214543 4560 scope.go:122] "RemoveContainer" containerID="2762fbbf89dedd226b280c56db50084b04658440d4ce0b507ac9a00ccca38ebc" Oct 30 21:45:20.214657 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:20.214310 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889945-kvfr9" Oct 30 21:45:20.216266 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:20.216241 4560 generic.go:356] "Generic (PLEG): container finished" podID="9efba11d-653c-47de-9eaf-dc250f12d18c" containerID="32a9401f562311c7b2adff2eed2206f78d3f869632910a1a1af42aa0f1da9ce5" exitCode=0 Oct 30 21:45:20.216461 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:20.216284 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ee6b3c-fxrpq" event={"ID":"9efba11d-653c-47de-9eaf-dc250f12d18c","Type":"ContainerDied","Data":"32a9401f562311c7b2adff2eed2206f78d3f869632910a1a1af42aa0f1da9ce5"} Oct 30 21:45:20.218401 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:20.218381 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" event={"ID":"2bb9a4d9-6480-470b-967f-96b99f9ff606","Type":"ContainerStarted","Data":"95ec0a84a1a56bce468788fb97dad5fe4d70f7ff4d155f9f804e617f90b63930"} Oct 30 21:45:20.218563 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:20.218547 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" event={"ID":"2bb9a4d9-6480-470b-967f-96b99f9ff606","Type":"ContainerStarted","Data":"d07b5227190a3aacfa6e5da86317a330cc2c6e79f005fe1e39e01da91bced79b"} Oct 30 21:45:20.510749 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:20.510517 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" podStartSLOduration=2.510501916 podStartE2EDuration="2.510501916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:20.399800912 +0000 UTC m=+1895.954500690" watchObservedRunningTime="2026-10-30 21:45:20.510501916 +0000 UTC m=+1896.065201694" Oct 30 21:45:20.514645 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:20.514620 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889945-kvfr9"] Oct 30 21:45:20.519052 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:20.519030 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889945-kvfr9"] Oct 30 21:45:20.610529 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:20.610493 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b0f43b9-38d3-4e37-a0ce-f987574873b9" path="/var/lib/kubelet/pods/0b0f43b9-38d3-4e37-a0ce-f987574873b9/volumes" Oct 30 21:45:20.611090 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:20.611074 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f301bcc-2a99-4dd3-bddc-b5af224d83f5" path="/var/lib/kubelet/pods/9f301bcc-2a99-4dd3-bddc-b5af224d83f5/volumes" Oct 30 21:45:20.615586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:20.615554 4560 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:45:20.615732 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:20.615617 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" podUID="b007cff0-0922-4fbf-b4fc-10aaf629a952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 21:45:20.849419 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:20.847894 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" Oct 30 21:45:20.849419 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:20.848085 4560 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:45:20.849419 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:20.848131 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" podUID="2bb9a4d9-6480-470b-967f-96b99f9ff606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 21:45:21.014416 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:21.012559 4560 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:45:21.014416 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:21.012625 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" podUID="aaf8da80-7e8e-40a3-8c88-33b2325ef2aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 21:45:21.630572 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:21.630488 4560 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:45:21.631591 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:21.631555 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" podUID="b007cff0-0922-4fbf-b4fc-10aaf629a952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 21:45:21.847137 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:21.847100 4560 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:45:21.847528 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:21.847498 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" podUID="2bb9a4d9-6480-470b-967f-96b99f9ff606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 21:45:22.462603 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:22.462564 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" podUID="a59a0fc3-8316-47eb-a42e-552d202b1ab7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:22.502366 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:22.468559 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" podUID="45a5bac0-28ab-4c78-856e-303f6d4522de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": context deadline exceeded" Oct 30 21:45:22.505129 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:22.505093 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" podUID="b80b0fba-3830-4a74-a724-aead35443b89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 30 21:45:22.519640 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:22.506302 4560 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:45:22.520169 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:22.511422 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" podUID="0ff8ae26-4364-4c4e-9285-721d1a486a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:22.520169 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:22.519186 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" podUID="1e044c02-0053-40e8-9f80-e806cd2d9ae5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:22.520169 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:22.519225 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" podUID="27df5dfe-47a4-4ebb-bd09-1ceef406600a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded" Oct 30 21:45:22.520503 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:22.520477 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" podUID="aaf8da80-7e8e-40a3-8c88-33b2325ef2aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 21:45:22.561332 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:22.559809 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" podUID="0155a02e-8df7-4847-b40f-b8d9e4bac8c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:22.581705 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:22.571234 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" podUID="4dee11ef-9647-47ee-a4b1-686680256542" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:22.682283 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:22.682245 4560 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:45:22.682559 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:22.682525 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" podUID="b007cff0-0922-4fbf-b4fc-10aaf629a952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 21:45:22.683080 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:22.683044 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" podUID="971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 21:45:22.702368 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:22.692219 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" podUID="90bfb1e4-9526-40f6-aed4-ee46c2d86471" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:22.750656 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:22.749849 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" podUID="9e2d7535-075d-4211-b814-35d6abff7605" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:22.754379 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:22.751514 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" podUID="8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 30 21:45:22.863889 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:22.863846 4560 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" podUID="45a5bac0-28ab-4c78-856e-303f6d4522de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:22.926536 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:22.872328 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ee6b3c-fxrpq" Oct 30 21:45:22.937404 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:22.937355 4560 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:45:22.937688 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:22.937657 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" podUID="2bb9a4d9-6480-470b-967f-96b99f9ff606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 21:45:23.030420 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.026791 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9efba11d-653c-47de-9eaf-dc250f12d18c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9efba11d-653c-47de-9eaf-dc250f12d18c-maas-api-openshift-ca\") pod \"9efba11d-653c-47de-9eaf-dc250f12d18c\" (UID: \"9efba11d-653c-47de-9eaf-dc250f12d18c\") " Oct 30 21:45:23.055365 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.030770 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9efba11d-653c-47de-9eaf-dc250f12d18c-kube-api-access-qnxjx\" (UniqueName: \"kubernetes.io/projected/9efba11d-653c-47de-9eaf-dc250f12d18c-kube-api-access-qnxjx\") pod \"9efba11d-653c-47de-9eaf-dc250f12d18c\" (UID: \"9efba11d-653c-47de-9eaf-dc250f12d18c\") " Oct 30 21:45:23.055869 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.055834 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9efba11d-653c-47de-9eaf-dc250f12d18c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9efba11d-653c-47de-9eaf-dc250f12d18c-maas-api-xks-ca\") pod \"9efba11d-653c-47de-9eaf-dc250f12d18c\" (UID: \"9efba11d-653c-47de-9eaf-dc250f12d18c\") " Oct 30 21:45:23.088365 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.050032 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9efba11d-653c-47de-9eaf-dc250f12d18c-maas-api-openshift-ca" pod "9efba11d-653c-47de-9eaf-dc250f12d18c" (UID: "9efba11d-653c-47de-9eaf-dc250f12d18c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:23.088639 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.059649 4560 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:45:23.088876 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.088842 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" podUID="aaf8da80-7e8e-40a3-8c88-33b2325ef2aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 21:45:23.137397 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.137355 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9efba11d-653c-47de-9eaf-dc250f12d18c-kube-api-access-qnxjx" pod "9efba11d-653c-47de-9eaf-dc250f12d18c" (UID: "9efba11d-653c-47de-9eaf-dc250f12d18c"). InnerVolumeSpecName "kube-api-access-qnxjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:23.180513 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.180473 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9efba11d-653c-47de-9eaf-dc250f12d18c-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:23.180775 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.180755 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnxjx\" (UniqueName: \"kubernetes.io/projected/9efba11d-653c-47de-9eaf-dc250f12d18c-kube-api-access-qnxjx\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:23.187211 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.187181 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9efba11d-653c-47de-9eaf-dc250f12d18c-maas-api-xks-ca" pod "9efba11d-653c-47de-9eaf-dc250f12d18c" (UID: "9efba11d-653c-47de-9eaf-dc250f12d18c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:23.294917 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.294822 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9efba11d-653c-47de-9eaf-dc250f12d18c-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:23.384364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.380437 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" podUID="e974bae1-bb18-46b0-a4ee-4c01d6455e32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:23.466205 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.466120 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" podUID="a59a0fc3-8316-47eb-a42e-552d202b1ab7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:23.469775 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.469751 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" Oct 30 21:45:23.473374 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.470395 4560 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" podUID="e974bae1-bb18-46b0-a4ee-4c01d6455e32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:23.529452 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.529412 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" podUID="e3281dbe-e8f9-4bab-9ae4-2279e79b7292" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 30 21:45:23.530155 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.529682 4560 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" podUID="1e044c02-0053-40e8-9f80-e806cd2d9ae5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": context deadline exceeded" Oct 30 21:45:23.530902 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.529725 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" podUID="45a5bac0-28ab-4c78-856e-303f6d4522de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:23.531257 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.529752 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" podUID="27df5dfe-47a4-4ebb-bd09-1ceef406600a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded" Oct 30 21:45:23.537366 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.529778 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" podUID="b80b0fba-3830-4a74-a724-aead35443b89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 30 21:45:23.537764 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.530805 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" podUID="0ff8ae26-4364-4c4e-9285-721d1a486a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:23.560414 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.530830 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" podUID="1e044c02-0053-40e8-9f80-e806cd2d9ae5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 30 21:45:23.560770 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.560730 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" podUID="0155a02e-8df7-4847-b40f-b8d9e4bac8c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:23.565724 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.531806 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" Oct 30 21:45:23.566506 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.566469 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" Oct 30 21:45:23.592751 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.566681 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" Oct 30 21:45:23.593069 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.581214 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" podUID="4dee11ef-9647-47ee-a4b1-686680256542" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:23.618881 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.618831 4560 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:45:23.619182 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.619132 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" podUID="b007cff0-0922-4fbf-b4fc-10aaf629a952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 21:45:23.633295 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.633270 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" Oct 30 21:45:23.633556 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.633538 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" Oct 30 21:45:23.636455 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.636432 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" Oct 30 21:45:23.636628 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.636614 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" Oct 30 21:45:23.716379 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.701134 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" podUID="971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:23.716379 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.701532 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" Oct 30 21:45:23.756375 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.750916 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" podUID="9e2d7535-075d-4211-b814-35d6abff7605" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:23.756375 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.751292 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" podUID="90bfb1e4-9526-40f6-aed4-ee46c2d86471" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Oct 30 21:45:23.756375 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.751630 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" Oct 30 21:45:23.756375 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.751687 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" Oct 30 21:45:23.786748 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.786700 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" podUID="8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:23.867969 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.867885 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" Oct 30 21:45:23.895364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.870912 4560 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:45:23.928723 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.896520 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ee6b3c-fxrpq" event={"ID":"9efba11d-653c-47de-9eaf-dc250f12d18c","Type":"ContainerDied","Data":"a3706c99a8391adb5a8a9e51ee1087ea73435bbefa25c0de2c432a447ac295ba"} Oct 30 21:45:23.981402 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.928952 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3706c99a8391adb5a8a9e51ee1087ea73435bbefa25c0de2c432a447ac295ba" Oct 30 21:45:23.981709 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.896597 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ee6b3c-fxrpq" Oct 30 21:45:23.981978 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.928579 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" podUID="2bb9a4d9-6480-470b-967f-96b99f9ff606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 21:45:23.982156 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:23.960570 4560 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" podUID="4dee11ef-9647-47ee-a4b1-686680256542" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:24.059865 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:24.059824 4560 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" podUID="b80b0fba-3830-4a74-a724-aead35443b89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:24.114364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:24.106971 4560 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:45:24.128664 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:24.114779 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" podUID="aaf8da80-7e8e-40a3-8c88-33b2325ef2aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 21:45:24.426421 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:24.401001 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" podUID="e974bae1-bb18-46b0-a4ee-4c01d6455e32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:24.516728 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:24.475436 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" podUID="a59a0fc3-8316-47eb-a42e-552d202b1ab7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 30 21:45:24.517643 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:24.517617 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" Oct 30 21:45:24.535371 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:24.532701 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" podUID="27df5dfe-47a4-4ebb-bd09-1ceef406600a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:24.535371 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:24.533004 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" podUID="e3281dbe-e8f9-4bab-9ae4-2279e79b7292" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:24.535371 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:24.533254 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" podUID="45a5bac0-28ab-4c78-856e-303f6d4522de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:24.535371 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:24.533645 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" Oct 30 21:45:24.547384 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:24.547321 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" podUID="b80b0fba-3830-4a74-a724-aead35443b89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 30 21:45:24.564645 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:24.564607 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" podUID="1e044c02-0053-40e8-9f80-e806cd2d9ae5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:24.568506 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:24.568473 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" podUID="0155a02e-8df7-4847-b40f-b8d9e4bac8c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded" Oct 30 21:45:24.583032 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:24.582984 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" podUID="0ff8ae26-4364-4c4e-9285-721d1a486a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:24.647191 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:24.647159 4560 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:45:24.672545 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:24.672492 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" podUID="b007cff0-0922-4fbf-b4fc-10aaf629a952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 21:45:24.672876 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:24.672847 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" podUID="4dee11ef-9647-47ee-a4b1-686680256542" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:24.793363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:24.790909 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" podUID="90bfb1e4-9526-40f6-aed4-ee46c2d86471" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:24.793363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:24.792088 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" podUID="9e2d7535-075d-4211-b814-35d6abff7605" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:45:24.871526 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:24.871481 4560 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:45:24.877602 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:24.877566 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" podUID="2bb9a4d9-6480-470b-967f-96b99f9ff606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 21:45:24.877830 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:24.876592 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" podUID="8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 30 21:45:24.919812 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:24.884438 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" podUID="971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 21:45:25.029454 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:25.021455 4560 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:45:25.029454 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:25.021530 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" podUID="aaf8da80-7e8e-40a3-8c88-33b2325ef2aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 21:45:25.459363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:25.458554 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aec7d28-wl4wq"] Oct 30 21:45:25.483449 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:25.478590 4560 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:45:25.483449 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:25.478649 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" podUID="b3a33e86-072f-458f-837d-d349fb6981da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:25.614391 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:25.574756 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" podUID="0155a02e-8df7-4847-b40f-b8d9e4bac8c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:25.623443 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:25.623397 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" podUID="1e044c02-0053-40e8-9f80-e806cd2d9ae5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:25.669379 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:25.669304 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" podUID="27df5dfe-47a4-4ebb-bd09-1ceef406600a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded" Oct 30 21:45:25.669988 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:25.669954 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" podUID="e3281dbe-e8f9-4bab-9ae4-2279e79b7292" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 30 21:45:25.670451 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:25.670429 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" podUID="45a5bac0-28ab-4c78-856e-303f6d4522de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": context deadline exceeded" Oct 30 21:45:25.699646 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:25.699616 4560 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:45:25.699891 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:25.699869 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" podUID="b007cff0-0922-4fbf-b4fc-10aaf629a952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 21:45:25.700009 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:25.674593 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" podUID="0ff8ae26-4364-4c4e-9285-721d1a486a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 21:45:25.700845 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:25.674548 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" podUID="4dee11ef-9647-47ee-a4b1-686680256542" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:25.731086 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:25.731051 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" podUID="a59a0fc3-8316-47eb-a42e-552d202b1ab7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 30 21:45:25.731625 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:25.731596 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" podUID="e974bae1-bb18-46b0-a4ee-4c01d6455e32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 21:45:25.732936 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:25.732915 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" podUID="b80b0fba-3830-4a74-a724-aead35443b89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 30 21:45:25.809992 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:25.807474 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" podUID="9e2d7535-075d-4211-b814-35d6abff7605" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:45:25.809992 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:25.807877 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" podUID="90bfb1e4-9526-40f6-aed4-ee46c2d86471" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Oct 30 21:45:25.900543 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:25.900491 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" podUID="8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 30 21:45:25.927165 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:25.927105 4560 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:45:25.935490 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:25.935428 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" podUID="f10eae6b-051d-4929-902f-abcd38146a15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:25.958674 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:25.958610 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" podUID="971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 21:45:26.126483 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:26.121301 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aec7d28-wl4wq"] Oct 30 21:45:26.216737 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:26.205232 4560 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:45:26.216737 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:26.205307 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" podUID="2bb9a4d9-6480-470b-967f-96b99f9ff606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 21:45:26.216737 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:26.205451 4560 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:45:26.216737 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:26.205482 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" podUID="aaf8da80-7e8e-40a3-8c88-33b2325ef2aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 21:45:26.216737 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:26.207967 4560 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" podUID="8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:26.226376 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:26.221477 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" Oct 30 21:45:26.316376 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:26.314460 4560 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:45:26.316376 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:26.314536 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" podUID="5d83608f-bcb0-4aab-b14e-29846561a3d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:26.341582 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:26.341447 4560 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:45:26.341582 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:26.341506 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" podUID="8e1ac4c8-c9d4-4419-8a37-cabedad5ad01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:26.418599 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:26.418510 4560 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:45:26.418842 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:26.418814 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" podUID="e71b8a5c-fcd2-44f0-86d4-984ab9512afd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:26.769450 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:26.647215 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" podUID="0155a02e-8df7-4847-b40f-b8d9e4bac8c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded" Oct 30 21:45:27.098419 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:26.779476 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" podUID="27df5dfe-47a4-4ebb-bd09-1ceef406600a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:27.098617 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:26.779535 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" podUID="45a5bac0-28ab-4c78-856e-303f6d4522de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:27.257399 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:26.779579 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" podUID="0ff8ae26-4364-4c4e-9285-721d1a486a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 21:45:27.257399 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:26.853400 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" podUID="4dee11ef-9647-47ee-a4b1-686680256542" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:27.257399 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:26.853473 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" podUID="e3281dbe-e8f9-4bab-9ae4-2279e79b7292" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 30 21:45:27.257399 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:26.888134 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" podUID="9e2d7535-075d-4211-b814-35d6abff7605" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:45:27.257399 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:26.888371 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" podUID="90bfb1e4-9526-40f6-aed4-ee46c2d86471" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Oct 30 21:45:27.257399 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:26.890476 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" podUID="a59a0fc3-8316-47eb-a42e-552d202b1ab7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 30 21:45:27.257399 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:26.890722 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" podUID="e974bae1-bb18-46b0-a4ee-4c01d6455e32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:27.257399 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:26.972289 4560 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:45:27.257399 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:27.229753 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" podUID="2bb9a4d9-6480-470b-967f-96b99f9ff606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 21:45:27.257399 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:27.036106 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" podUID="8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 30 21:45:27.257399 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:26.999088 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" podUID="971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 21:45:27.257399 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:27.001831 4560 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:45:27.257399 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:27.230310 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" podUID="b007cff0-0922-4fbf-b4fc-10aaf629a952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 21:45:27.257399 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:27.032031 4560 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:45:27.257399 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:27.230390 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" podUID="aaf8da80-7e8e-40a3-8c88-33b2325ef2aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 21:45:27.292360 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:27.258477 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" podUID="1e044c02-0053-40e8-9f80-e806cd2d9ae5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:27.292360 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:26.770322 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" podUID="b80b0fba-3830-4a74-a724-aead35443b89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:27.601606 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:27.601564 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a7da446-6981-4258-aca5-affa0cc7bb41" path="/var/lib/kubelet/pods/6a7da446-6981-4258-aca5-affa0cc7bb41/volumes" Oct 30 21:45:27.629052 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:27.629018 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" Oct 30 21:45:27.652535 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:27.652499 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" Oct 30 21:45:27.654421 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:27.615737 4560 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:45:27.655353 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:27.655311 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" podUID="b007cff0-0922-4fbf-b4fc-10aaf629a952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 21:45:27.681468 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:27.681421 4560 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" podUID="0ff8ae26-4364-4c4e-9285-721d1a486a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:27.847921 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:27.846784 4560 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:45:27.847921 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:27.846841 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" podUID="2bb9a4d9-6480-470b-967f-96b99f9ff606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 21:45:27.951381 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:27.951270 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4327b99b-l5szt"] Oct 30 21:45:27.995314 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:27.995280 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4327b99b-l5szt"] Oct 30 21:45:28.014820 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:28.014691 4560 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:45:28.014820 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:28.014750 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" podUID="aaf8da80-7e8e-40a3-8c88-33b2325ef2aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 21:45:28.099010 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:28.098969 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" podUID="45a5bac0-28ab-4c78-856e-303f6d4522de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:28.099570 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:28.099537 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" podUID="0155a02e-8df7-4847-b40f-b8d9e4bac8c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:28.106429 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:28.106403 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" podUID="27df5dfe-47a4-4ebb-bd09-1ceef406600a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded" Oct 30 21:45:28.190481 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:28.190437 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-329ab292-q5xcq"] Oct 30 21:45:28.283366 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:28.230471 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" podUID="971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:28.339365 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:28.273432 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" podUID="90bfb1e4-9526-40f6-aed4-ee46c2d86471" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:28.486370 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:28.273476 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" podUID="0ff8ae26-4364-4c4e-9285-721d1a486a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 21:45:28.486370 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:28.273497 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" podUID="4dee11ef-9647-47ee-a4b1-686680256542" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Oct 30 21:45:28.486370 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:28.280491 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" podUID="e974bae1-bb18-46b0-a4ee-4c01d6455e32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:28.486370 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:28.280538 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" podUID="a59a0fc3-8316-47eb-a42e-552d202b1ab7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 30 21:45:28.486370 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:28.280568 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" podUID="8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:28.486370 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:28.280619 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" podUID="1e044c02-0053-40e8-9f80-e806cd2d9ae5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:28.486370 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:28.373039 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" Oct 30 21:45:28.486370 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:28.459607 4560 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" podUID="27df5dfe-47a4-4ebb-bd09-1ceef406600a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:28.487441 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:28.487402 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" podUID="9e2d7535-075d-4211-b814-35d6abff7605" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:28.787469 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:28.787424 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" podUID="b80b0fba-3830-4a74-a724-aead35443b89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:28.923366 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:28.812999 4560 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" podUID="9e2d7535-075d-4211-b814-35d6abff7605" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:29.097027 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:29.096912 4560 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:45:29.107366 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:29.097294 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" podUID="2bb9a4d9-6480-470b-967f-96b99f9ff606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 21:45:29.209505 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:29.107776 4560 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:45:29.209785 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:29.209756 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" podUID="b007cff0-0922-4fbf-b4fc-10aaf629a952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 21:45:29.209901 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:29.203796 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" podUID="45a5bac0-28ab-4c78-856e-303f6d4522de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": context deadline exceeded" Oct 30 21:45:29.211747 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:29.113434 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" podUID="27df5dfe-47a4-4ebb-bd09-1ceef406600a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded" Oct 30 21:45:29.212185 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:29.203849 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" podUID="0155a02e-8df7-4847-b40f-b8d9e4bac8c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded" Oct 30 21:45:29.280521 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:29.280245 4560 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:45:29.280521 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:29.280316 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" podUID="aaf8da80-7e8e-40a3-8c88-33b2325ef2aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 21:45:29.377456 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:29.366679 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" podUID="971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 21:45:29.396909 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:29.387011 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" podUID="1e044c02-0053-40e8-9f80-e806cd2d9ae5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:29.397629 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:29.397589 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" podUID="4dee11ef-9647-47ee-a4b1-686680256542" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Oct 30 21:45:29.398175 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:29.398149 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" podUID="8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 30 21:45:29.398596 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:29.388013 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49da1004-a76f-46f5-925d-a6a17be6d6e7" path="/var/lib/kubelet/pods/49da1004-a76f-46f5-925d-a6a17be6d6e7/volumes" Oct 30 21:45:29.399305 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:29.399285 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" Oct 30 21:45:29.399514 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:29.399499 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-329ab292-q5xcq"] Oct 30 21:45:29.403065 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:29.403046 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" Oct 30 21:45:29.404150 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:29.404118 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" Oct 30 21:45:29.405266 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:29.405249 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" Oct 30 21:45:29.413357 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:29.413322 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" Oct 30 21:45:29.413591 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:29.413577 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" Oct 30 21:45:29.430366 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:29.426677 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c9ee410-qbfhf"] Oct 30 21:45:29.524500 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:29.520281 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" Oct 30 21:45:29.558362 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:29.524938 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" Oct 30 21:45:29.558362 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:29.549016 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" Oct 30 21:45:29.558362 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:29.553914 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c9ee410-qbfhf"] Oct 30 21:45:29.580554 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:29.580026 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" Oct 30 21:45:29.585286 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:29.582481 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" Oct 30 21:45:29.588806 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:29.587150 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" Oct 30 21:45:29.615366 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:29.615118 4560 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:45:29.615366 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:29.615164 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" podUID="b007cff0-0922-4fbf-b4fc-10aaf629a952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 21:45:29.843492 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:29.843461 4560 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:45:29.843760 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:29.843732 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" podUID="2bb9a4d9-6480-470b-967f-96b99f9ff606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 21:45:30.013659 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:30.013610 4560 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:45:30.014222 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:30.014191 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" podUID="aaf8da80-7e8e-40a3-8c88-33b2325ef2aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 21:45:30.581850 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:30.581814 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c851243-770c-4819-abb6-b19588f0fffe" path="/var/lib/kubelet/pods/0c851243-770c-4819-abb6-b19588f0fffe/volumes" Oct 30 21:45:30.582764 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:30.582742 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1f388b2-e1b0-4df6-b553-883a63f3401b" path="/var/lib/kubelet/pods/b1f388b2-e1b0-4df6-b553-883a63f3401b/volumes" Oct 30 21:45:30.710891 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:30.710858 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d0e8d90-8g7xn"] Oct 30 21:45:30.711815 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:30.711790 4560 cpu_manager.go:414] "containerMap: removing container" podUID="9efba11d-653c-47de-9eaf-dc250f12d18c" containerName="revoke-keys" Oct 30 21:45:30.711947 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:30.711933 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9efba11d-653c-47de-9eaf-dc250f12d18c" containerName="revoke-keys" Oct 30 21:45:30.712060 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:30.712032 4560 cpu_manager.go:414] "containerMap: removing container" podUID="0b0f43b9-38d3-4e37-a0ce-f987574873b9" containerName="cleanup" Oct 30 21:45:30.712195 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:30.712180 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b0f43b9-38d3-4e37-a0ce-f987574873b9" containerName="cleanup" Oct 30 21:45:30.712408 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:30.712392 4560 cpu_manager.go:414] "containerMap: removing container" podUID="0b0f43b9-38d3-4e37-a0ce-f987574873b9" containerName="cleanup" Oct 30 21:45:30.712551 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:30.712536 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b0f43b9-38d3-4e37-a0ce-f987574873b9" containerName="cleanup" Oct 30 21:45:30.712821 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:30.712804 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b0f43b9-38d3-4e37-a0ce-f987574873b9" containerName="cleanup" Oct 30 21:45:30.712933 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:30.712921 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b0f43b9-38d3-4e37-a0ce-f987574873b9" containerName="cleanup" Oct 30 21:45:30.713021 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:30.713011 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="9efba11d-653c-47de-9eaf-dc250f12d18c" containerName="revoke-keys" Oct 30 21:45:30.713109 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:30.713099 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b0f43b9-38d3-4e37-a0ce-f987574873b9" containerName="cleanup" Oct 30 21:45:30.732381 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:30.732357 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0e8d90-8g7xn" Oct 30 21:45:30.795728 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:30.795694 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d0e8d90-8g7xn"] Oct 30 21:45:30.807752 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:30.807719 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:45:30.815406 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:30.808009 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:45:30.815545 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:30.808126 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:45:30.815635 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:30.808217 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:45:30.824406 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:30.820475 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/567f946c-fb5c-4798-b11c-bae6723f3d89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d0e8d90-8g7xn\" (UID: \"567f946c-fb5c-4798-b11c-bae6723f3d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0e8d90-8g7xn" Oct 30 21:45:30.824406 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:30.820758 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/567f946c-fb5c-4798-b11c-bae6723f3d89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d0e8d90-8g7xn\" (UID: \"567f946c-fb5c-4798-b11c-bae6723f3d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0e8d90-8g7xn" Oct 30 21:45:30.824406 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:30.820867 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4jdj\" (UniqueName: \"kubernetes.io/projected/567f946c-fb5c-4798-b11c-bae6723f3d89-kube-api-access-g4jdj\") pod \"maas-api-revoke-sub-7d0e8d90-8g7xn\" (UID: \"567f946c-fb5c-4798-b11c-bae6723f3d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0e8d90-8g7xn" Oct 30 21:45:30.870072 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:30.869993 4560 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:45:30.870312 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:30.870283 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" podUID="2bb9a4d9-6480-470b-967f-96b99f9ff606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 21:45:30.941980 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:30.941937 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/567f946c-fb5c-4798-b11c-bae6723f3d89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d0e8d90-8g7xn\" (UID: \"567f946c-fb5c-4798-b11c-bae6723f3d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0e8d90-8g7xn" Oct 30 21:45:30.942175 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:30.942059 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4jdj\" (UniqueName: \"kubernetes.io/projected/567f946c-fb5c-4798-b11c-bae6723f3d89-kube-api-access-g4jdj\") pod \"maas-api-revoke-sub-7d0e8d90-8g7xn\" (UID: \"567f946c-fb5c-4798-b11c-bae6723f3d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0e8d90-8g7xn" Oct 30 21:45:30.942175 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:30.942134 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/567f946c-fb5c-4798-b11c-bae6723f3d89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d0e8d90-8g7xn\" (UID: \"567f946c-fb5c-4798-b11c-bae6723f3d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0e8d90-8g7xn" Oct 30 21:45:30.943054 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:30.943027 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/567f946c-fb5c-4798-b11c-bae6723f3d89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d0e8d90-8g7xn\" (UID: \"567f946c-fb5c-4798-b11c-bae6723f3d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0e8d90-8g7xn" Oct 30 21:45:30.950847 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:30.950825 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/567f946c-fb5c-4798-b11c-bae6723f3d89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d0e8d90-8g7xn\" (UID: \"567f946c-fb5c-4798-b11c-bae6723f3d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0e8d90-8g7xn" Oct 30 21:45:31.002298 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:31.002264 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4jdj\" (UniqueName: \"kubernetes.io/projected/567f946c-fb5c-4798-b11c-bae6723f3d89-kube-api-access-g4jdj\") pod \"maas-api-revoke-sub-7d0e8d90-8g7xn\" (UID: \"567f946c-fb5c-4798-b11c-bae6723f3d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0e8d90-8g7xn" Oct 30 21:45:31.174121 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:31.174039 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0e8d90-8g7xn" Oct 30 21:45:31.299192 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:31.297943 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" Oct 30 21:45:31.300127 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:31.299676 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" Oct 30 21:45:31.300848 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:31.300794 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" Oct 30 21:45:31.313949 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:31.312580 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" Oct 30 21:45:31.313949 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:31.313736 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" Oct 30 21:45:31.314526 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:31.314480 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" Oct 30 21:45:31.577077 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:31.577044 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d0e8d90-8g7xn"] Oct 30 21:45:32.184994 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.184955 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70cf0de4-b9pvm"] Oct 30 21:45:32.186327 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.186290 4560 cpu_manager.go:414] "containerMap: removing container" podUID="0b0f43b9-38d3-4e37-a0ce-f987574873b9" containerName="cleanup" Oct 30 21:45:32.186327 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.186327 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b0f43b9-38d3-4e37-a0ce-f987574873b9" containerName="cleanup" Oct 30 21:45:32.191767 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.191742 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cf0de4-b9pvm" Oct 30 21:45:32.215517 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.215489 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aaa2a11-2803-4d9d-9bf2-75175375cb8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70cf0de4-b9pvm\" (UID: \"7aaa2a11-2803-4d9d-9bf2-75175375cb8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cf0de4-b9pvm" Oct 30 21:45:32.215723 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.215699 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cpnr\" (UniqueName: \"kubernetes.io/projected/7aaa2a11-2803-4d9d-9bf2-75175375cb8f-kube-api-access-2cpnr\") pod \"maas-api-revoke-sub-70cf0de4-b9pvm\" (UID: \"7aaa2a11-2803-4d9d-9bf2-75175375cb8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cf0de4-b9pvm" Oct 30 21:45:32.215886 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.215848 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aaa2a11-2803-4d9d-9bf2-75175375cb8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70cf0de4-b9pvm\" (UID: \"7aaa2a11-2803-4d9d-9bf2-75175375cb8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cf0de4-b9pvm" Oct 30 21:45:32.255051 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.255006 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70cf0de4-b9pvm"] Oct 30 21:45:32.307529 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.307482 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" Oct 30 21:45:32.307916 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.307888 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" Oct 30 21:45:32.308807 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.308787 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" Oct 30 21:45:32.316904 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.316880 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aaa2a11-2803-4d9d-9bf2-75175375cb8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70cf0de4-b9pvm\" (UID: \"7aaa2a11-2803-4d9d-9bf2-75175375cb8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cf0de4-b9pvm" Oct 30 21:45:32.317032 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.316987 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cpnr\" (UniqueName: \"kubernetes.io/projected/7aaa2a11-2803-4d9d-9bf2-75175375cb8f-kube-api-access-2cpnr\") pod \"maas-api-revoke-sub-70cf0de4-b9pvm\" (UID: \"7aaa2a11-2803-4d9d-9bf2-75175375cb8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cf0de4-b9pvm" Oct 30 21:45:32.317094 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.317033 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aaa2a11-2803-4d9d-9bf2-75175375cb8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70cf0de4-b9pvm\" (UID: \"7aaa2a11-2803-4d9d-9bf2-75175375cb8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cf0de4-b9pvm" Oct 30 21:45:32.317620 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.317596 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aaa2a11-2803-4d9d-9bf2-75175375cb8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70cf0de4-b9pvm\" (UID: \"7aaa2a11-2803-4d9d-9bf2-75175375cb8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cf0de4-b9pvm" Oct 30 21:45:32.319428 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.319407 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aaa2a11-2803-4d9d-9bf2-75175375cb8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70cf0de4-b9pvm\" (UID: \"7aaa2a11-2803-4d9d-9bf2-75175375cb8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cf0de4-b9pvm" Oct 30 21:45:32.383783 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.383749 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cpnr\" (UniqueName: \"kubernetes.io/projected/7aaa2a11-2803-4d9d-9bf2-75175375cb8f-kube-api-access-2cpnr\") pod \"maas-api-revoke-sub-70cf0de4-b9pvm\" (UID: \"7aaa2a11-2803-4d9d-9bf2-75175375cb8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cf0de4-b9pvm" Oct 30 21:45:32.408420 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.408387 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv"] Oct 30 21:45:32.413465 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.413442 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" Oct 30 21:45:32.503289 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.503253 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv"] Oct 30 21:45:32.506955 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.506922 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cf0de4-b9pvm" Oct 30 21:45:32.519167 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.519131 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57d2540b-40a0-4326-a4af-57466f9abbae-workload-certs\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv\" (UID: \"57d2540b-40a0-4326-a4af-57466f9abbae\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" Oct 30 21:45:32.519318 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.519175 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d2540b-40a0-4326-a4af-57466f9abbae-workload-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv\" (UID: \"57d2540b-40a0-4326-a4af-57466f9abbae\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" Oct 30 21:45:32.519318 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.519263 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57d2540b-40a0-4326-a4af-57466f9abbae-istio-data\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv\" (UID: \"57d2540b-40a0-4326-a4af-57466f9abbae\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" Oct 30 21:45:32.519485 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.519319 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57d2540b-40a0-4326-a4af-57466f9abbae-istio-token\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv\" (UID: \"57d2540b-40a0-4326-a4af-57466f9abbae\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" Oct 30 21:45:32.519485 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.519382 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d2540b-40a0-4326-a4af-57466f9abbae-credential-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv\" (UID: \"57d2540b-40a0-4326-a4af-57466f9abbae\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" Oct 30 21:45:32.519630 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.519487 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57d2540b-40a0-4326-a4af-57466f9abbae-istio-envoy\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv\" (UID: \"57d2540b-40a0-4326-a4af-57466f9abbae\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" Oct 30 21:45:32.519630 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.519513 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs8r7\" (UniqueName: \"kubernetes.io/projected/57d2540b-40a0-4326-a4af-57466f9abbae-kube-api-access-qs8r7\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv\" (UID: \"57d2540b-40a0-4326-a4af-57466f9abbae\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" Oct 30 21:45:32.519630 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.519546 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57d2540b-40a0-4326-a4af-57466f9abbae-istiod-ca-cert\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv\" (UID: \"57d2540b-40a0-4326-a4af-57466f9abbae\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" Oct 30 21:45:32.519630 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.519568 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57d2540b-40a0-4326-a4af-57466f9abbae-istio-podinfo\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv\" (UID: \"57d2540b-40a0-4326-a4af-57466f9abbae\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" Oct 30 21:45:32.520288 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.520243 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0e8d90-8g7xn" event={"ID":"567f946c-fb5c-4798-b11c-bae6723f3d89","Type":"ContainerStarted","Data":"eec5a9005f5115f1d5e8226609ddda9c5fa6955aad6bd5a5d9e8fababb1fd0cd"} Oct 30 21:45:32.520358 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.520306 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0e8d90-8g7xn" event={"ID":"567f946c-fb5c-4798-b11c-bae6723f3d89","Type":"ContainerStarted","Data":"2c1ba36098f1eb57ebbe29f2549bf9510672075cb17639c02b95391e6c6b4481"} Oct 30 21:45:32.621030 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.621000 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57d2540b-40a0-4326-a4af-57466f9abbae-istio-token\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv\" (UID: \"57d2540b-40a0-4326-a4af-57466f9abbae\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" Oct 30 21:45:32.621204 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.621192 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d2540b-40a0-4326-a4af-57466f9abbae-credential-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv\" (UID: \"57d2540b-40a0-4326-a4af-57466f9abbae\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" Oct 30 21:45:32.623395 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.623374 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57d2540b-40a0-4326-a4af-57466f9abbae-istio-envoy\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv\" (UID: \"57d2540b-40a0-4326-a4af-57466f9abbae\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" Oct 30 21:45:32.623562 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.623545 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs8r7\" (UniqueName: \"kubernetes.io/projected/57d2540b-40a0-4326-a4af-57466f9abbae-kube-api-access-qs8r7\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv\" (UID: \"57d2540b-40a0-4326-a4af-57466f9abbae\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" Oct 30 21:45:32.623686 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.623674 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57d2540b-40a0-4326-a4af-57466f9abbae-istiod-ca-cert\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv\" (UID: \"57d2540b-40a0-4326-a4af-57466f9abbae\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" Oct 30 21:45:32.623762 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.623752 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57d2540b-40a0-4326-a4af-57466f9abbae-istio-podinfo\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv\" (UID: \"57d2540b-40a0-4326-a4af-57466f9abbae\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" Oct 30 21:45:32.623884 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.623876 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57d2540b-40a0-4326-a4af-57466f9abbae-workload-certs\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv\" (UID: \"57d2540b-40a0-4326-a4af-57466f9abbae\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" Oct 30 21:45:32.623947 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.623939 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d2540b-40a0-4326-a4af-57466f9abbae-workload-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv\" (UID: \"57d2540b-40a0-4326-a4af-57466f9abbae\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" Oct 30 21:45:32.624049 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.624040 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57d2540b-40a0-4326-a4af-57466f9abbae-istio-data\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv\" (UID: \"57d2540b-40a0-4326-a4af-57466f9abbae\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" Oct 30 21:45:32.624525 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.624510 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57d2540b-40a0-4326-a4af-57466f9abbae-istio-data\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv\" (UID: \"57d2540b-40a0-4326-a4af-57466f9abbae\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" Oct 30 21:45:32.625470 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.625351 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d2540b-40a0-4326-a4af-57466f9abbae-credential-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv\" (UID: \"57d2540b-40a0-4326-a4af-57466f9abbae\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" Oct 30 21:45:32.626025 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.626003 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57d2540b-40a0-4326-a4af-57466f9abbae-istiod-ca-cert\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv\" (UID: \"57d2540b-40a0-4326-a4af-57466f9abbae\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" Oct 30 21:45:32.626957 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.626620 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d2540b-40a0-4326-a4af-57466f9abbae-workload-socket\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv\" (UID: \"57d2540b-40a0-4326-a4af-57466f9abbae\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" Oct 30 21:45:32.627172 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.627152 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57d2540b-40a0-4326-a4af-57466f9abbae-workload-certs\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv\" (UID: \"57d2540b-40a0-4326-a4af-57466f9abbae\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" Oct 30 21:45:32.630895 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.630014 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57d2540b-40a0-4326-a4af-57466f9abbae-istio-envoy\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv\" (UID: \"57d2540b-40a0-4326-a4af-57466f9abbae\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" Oct 30 21:45:32.641517 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.639157 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57d2540b-40a0-4326-a4af-57466f9abbae-istio-podinfo\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv\" (UID: \"57d2540b-40a0-4326-a4af-57466f9abbae\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" Oct 30 21:45:32.694064 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.694020 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs8r7\" (UniqueName: \"kubernetes.io/projected/57d2540b-40a0-4326-a4af-57466f9abbae-kube-api-access-qs8r7\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv\" (UID: \"57d2540b-40a0-4326-a4af-57466f9abbae\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" Oct 30 21:45:32.701862 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.701834 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57d2540b-40a0-4326-a4af-57466f9abbae-istio-token\") pod \"e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv\" (UID: \"57d2540b-40a0-4326-a4af-57466f9abbae\") " pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" Oct 30 21:45:32.727931 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.727907 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" Oct 30 21:45:32.931233 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.931183 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0e8d90-8g7xn" podStartSLOduration=2.931169038 podStartE2EDuration="2.931169038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:32.716203529 +0000 UTC m=+1908.270903307" watchObservedRunningTime="2026-10-30 21:45:32.931169038 +0000 UTC m=+1908.485868814" Oct 30 21:45:32.933253 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:32.933203 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70cf0de4-b9pvm"] Oct 30 21:45:32.937569 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:45:32.937542 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aaa2a11_2803_4d9d_9bf2_75175375cb8f.slice/crio-54a152f6c67f19aebc4d98702d20e933f35dc52531fe23bae4a703fb8ecb36ab WatchSource:0}: Error finding container 54a152f6c67f19aebc4d98702d20e933f35dc52531fe23bae4a703fb8ecb36ab: Status 404 returned error can't find the container with id 54a152f6c67f19aebc4d98702d20e933f35dc52531fe23bae4a703fb8ecb36ab Oct 30 21:45:33.144898 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:33.144870 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv"] Oct 30 21:45:33.146142 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:45:33.146108 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57d2540b_40a0_4326_a4af_57466f9abbae.slice/crio-f2569acb84c371757485d7c7adb6ea2f9cd67daf696aca1c948dcb2c5ffb0edf WatchSource:0}: Error finding container f2569acb84c371757485d7c7adb6ea2f9cd67daf696aca1c948dcb2c5ffb0edf: Status 404 returned error can't find the container with id f2569acb84c371757485d7c7adb6ea2f9cd67daf696aca1c948dcb2c5ffb0edf Oct 30 21:45:33.148715 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:33.148682 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:45:33.148804 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:33.148749 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:45:33.148804 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:33.148779 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:45:33.532910 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:33.532876 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" event={"ID":"57d2540b-40a0-4326-a4af-57466f9abbae","Type":"ContainerStarted","Data":"b04c6444b1f6db659f1972b7786433757c02ecd83449a2cd3a2120d439607b8c"} Oct 30 21:45:33.533530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:33.533487 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" event={"ID":"57d2540b-40a0-4326-a4af-57466f9abbae","Type":"ContainerStarted","Data":"f2569acb84c371757485d7c7adb6ea2f9cd67daf696aca1c948dcb2c5ffb0edf"} Oct 30 21:45:33.534739 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:33.534715 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cf0de4-b9pvm" event={"ID":"7aaa2a11-2803-4d9d-9bf2-75175375cb8f","Type":"ContainerStarted","Data":"32bf939d57ac2e126f84b717ea48117229fdf9498e669754183332fda92c3588"} Oct 30 21:45:33.534832 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:33.534758 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cf0de4-b9pvm" event={"ID":"7aaa2a11-2803-4d9d-9bf2-75175375cb8f","Type":"ContainerStarted","Data":"54a152f6c67f19aebc4d98702d20e933f35dc52531fe23bae4a703fb8ecb36ab"} Oct 30 21:45:33.536374 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:33.536333 4560 generic.go:356] "Generic (PLEG): container finished" podID="567f946c-fb5c-4798-b11c-bae6723f3d89" containerID="eec5a9005f5115f1d5e8226609ddda9c5fa6955aad6bd5a5d9e8fababb1fd0cd" exitCode=0 Oct 30 21:45:33.536476 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:33.536382 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0e8d90-8g7xn" event={"ID":"567f946c-fb5c-4798-b11c-bae6723f3d89","Type":"ContainerDied","Data":"eec5a9005f5115f1d5e8226609ddda9c5fa6955aad6bd5a5d9e8fababb1fd0cd"} Oct 30 21:45:33.652753 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:33.652702 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" podStartSLOduration=1.652689273 podStartE2EDuration="1.652689273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:33.649095763 +0000 UTC m=+1909.203795531" watchObservedRunningTime="2026-10-30 21:45:33.652689273 +0000 UTC m=+1909.207389049" Oct 30 21:45:33.728574 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:33.728539 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" Oct 30 21:45:33.764259 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:33.764206 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cf0de4-b9pvm" podStartSLOduration=1.764193799 podStartE2EDuration="1.764193799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:33.761976709 +0000 UTC m=+1909.316676484" watchObservedRunningTime="2026-10-30 21:45:33.764193799 +0000 UTC m=+1909.318893576" Oct 30 21:45:33.991219 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:33.991188 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a878660f-q4pkx"] Oct 30 21:45:33.995815 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:33.995795 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a878660f-q4pkx" Oct 30 21:45:34.055361 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:34.055250 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a878660f-q4pkx"] Oct 30 21:45:34.057362 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:34.057278 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ace52bf5-0ea8-4d12-94b9-e234c93f8720-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a878660f-q4pkx\" (UID: \"ace52bf5-0ea8-4d12-94b9-e234c93f8720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a878660f-q4pkx" Oct 30 21:45:34.057362 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:34.057313 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ace52bf5-0ea8-4d12-94b9-e234c93f8720-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a878660f-q4pkx\" (UID: \"ace52bf5-0ea8-4d12-94b9-e234c93f8720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a878660f-q4pkx" Oct 30 21:45:34.057362 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:34.057356 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6cmf\" (UniqueName: \"kubernetes.io/projected/ace52bf5-0ea8-4d12-94b9-e234c93f8720-kube-api-access-w6cmf\") pod \"maas-api-revoke-sub-a878660f-q4pkx\" (UID: \"ace52bf5-0ea8-4d12-94b9-e234c93f8720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a878660f-q4pkx" Oct 30 21:45:34.158509 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:34.158476 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ace52bf5-0ea8-4d12-94b9-e234c93f8720-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a878660f-q4pkx\" (UID: \"ace52bf5-0ea8-4d12-94b9-e234c93f8720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a878660f-q4pkx" Oct 30 21:45:34.158690 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:34.158518 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ace52bf5-0ea8-4d12-94b9-e234c93f8720-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a878660f-q4pkx\" (UID: \"ace52bf5-0ea8-4d12-94b9-e234c93f8720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a878660f-q4pkx" Oct 30 21:45:34.158690 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:34.158546 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6cmf\" (UniqueName: \"kubernetes.io/projected/ace52bf5-0ea8-4d12-94b9-e234c93f8720-kube-api-access-w6cmf\") pod \"maas-api-revoke-sub-a878660f-q4pkx\" (UID: \"ace52bf5-0ea8-4d12-94b9-e234c93f8720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a878660f-q4pkx" Oct 30 21:45:34.159156 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:34.159126 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ace52bf5-0ea8-4d12-94b9-e234c93f8720-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a878660f-q4pkx\" (UID: \"ace52bf5-0ea8-4d12-94b9-e234c93f8720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a878660f-q4pkx" Oct 30 21:45:34.160860 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:34.160840 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ace52bf5-0ea8-4d12-94b9-e234c93f8720-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a878660f-q4pkx\" (UID: \"ace52bf5-0ea8-4d12-94b9-e234c93f8720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a878660f-q4pkx" Oct 30 21:45:34.221328 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:34.221252 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6cmf\" (UniqueName: \"kubernetes.io/projected/ace52bf5-0ea8-4d12-94b9-e234c93f8720-kube-api-access-w6cmf\") pod \"maas-api-revoke-sub-a878660f-q4pkx\" (UID: \"ace52bf5-0ea8-4d12-94b9-e234c93f8720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a878660f-q4pkx" Oct 30 21:45:34.345086 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:34.345050 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a878660f-q4pkx" Oct 30 21:45:34.565850 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:34.565817 4560 generic.go:356] "Generic (PLEG): container finished" podID="7aaa2a11-2803-4d9d-9bf2-75175375cb8f" containerID="32bf939d57ac2e126f84b717ea48117229fdf9498e669754183332fda92c3588" exitCode=0 Oct 30 21:45:34.566636 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:34.566611 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cf0de4-b9pvm" event={"ID":"7aaa2a11-2803-4d9d-9bf2-75175375cb8f","Type":"ContainerDied","Data":"32bf939d57ac2e126f84b717ea48117229fdf9498e669754183332fda92c3588"} Oct 30 21:45:34.760763 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:34.760729 4560 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 21:45:34.760960 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:34.760942 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" podUID="57d2540b-40a0-4326-a4af-57466f9abbae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 21:45:34.763103 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:34.763077 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a878660f-q4pkx"] Oct 30 21:45:34.793306 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:34.793193 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0e8d90-8g7xn" Oct 30 21:45:34.876181 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:34.876106 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/567f946c-fb5c-4798-b11c-bae6723f3d89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/567f946c-fb5c-4798-b11c-bae6723f3d89-maas-api-openshift-ca\") pod \"567f946c-fb5c-4798-b11c-bae6723f3d89\" (UID: \"567f946c-fb5c-4798-b11c-bae6723f3d89\") " Oct 30 21:45:34.876181 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:34.876158 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/567f946c-fb5c-4798-b11c-bae6723f3d89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/567f946c-fb5c-4798-b11c-bae6723f3d89-maas-api-xks-ca\") pod \"567f946c-fb5c-4798-b11c-bae6723f3d89\" (UID: \"567f946c-fb5c-4798-b11c-bae6723f3d89\") " Oct 30 21:45:34.876440 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:34.876419 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/567f946c-fb5c-4798-b11c-bae6723f3d89-kube-api-access-g4jdj\" (UniqueName: \"kubernetes.io/projected/567f946c-fb5c-4798-b11c-bae6723f3d89-kube-api-access-g4jdj\") pod \"567f946c-fb5c-4798-b11c-bae6723f3d89\" (UID: \"567f946c-fb5c-4798-b11c-bae6723f3d89\") " Oct 30 21:45:34.876551 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:34.876523 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/567f946c-fb5c-4798-b11c-bae6723f3d89-maas-api-openshift-ca" pod "567f946c-fb5c-4798-b11c-bae6723f3d89" (UID: "567f946c-fb5c-4798-b11c-bae6723f3d89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:34.876873 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:34.876849 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/567f946c-fb5c-4798-b11c-bae6723f3d89-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:34.962700 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:34.962665 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/567f946c-fb5c-4798-b11c-bae6723f3d89-maas-api-xks-ca" pod "567f946c-fb5c-4798-b11c-bae6723f3d89" (UID: "567f946c-fb5c-4798-b11c-bae6723f3d89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:34.962880 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:34.962858 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/567f946c-fb5c-4798-b11c-bae6723f3d89-kube-api-access-g4jdj" pod "567f946c-fb5c-4798-b11c-bae6723f3d89" (UID: "567f946c-fb5c-4798-b11c-bae6723f3d89"). InnerVolumeSpecName "kube-api-access-g4jdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:34.978237 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:34.978204 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4jdj\" (UniqueName: \"kubernetes.io/projected/567f946c-fb5c-4798-b11c-bae6723f3d89-kube-api-access-g4jdj\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:34.978392 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:34.978241 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/567f946c-fb5c-4798-b11c-bae6723f3d89-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:35.575647 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:35.575620 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0e8d90-8g7xn" Oct 30 21:45:35.576059 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:35.575651 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0e8d90-8g7xn" event={"ID":"567f946c-fb5c-4798-b11c-bae6723f3d89","Type":"ContainerDied","Data":"2c1ba36098f1eb57ebbe29f2549bf9510672075cb17639c02b95391e6c6b4481"} Oct 30 21:45:35.576059 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:35.575691 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c1ba36098f1eb57ebbe29f2549bf9510672075cb17639c02b95391e6c6b4481" Oct 30 21:45:35.577498 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:35.577472 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a878660f-q4pkx" event={"ID":"ace52bf5-0ea8-4d12-94b9-e234c93f8720","Type":"ContainerStarted","Data":"7bb4dc5ee6d7c667bb8b520957a0c46fb0be949333a9ad9ce2db6fee1b83872a"} Oct 30 21:45:35.577671 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:35.577519 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a878660f-q4pkx" event={"ID":"ace52bf5-0ea8-4d12-94b9-e234c93f8720","Type":"ContainerStarted","Data":"36f629ad6eb41c155557b8bd95c39eeff63f4a716dbbcbe20d6c1e8d89f6df5d"} Oct 30 21:45:35.735365 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:35.730766 4560 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 21:45:35.735365 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:35.730825 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" podUID="57d2540b-40a0-4326-a4af-57466f9abbae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 21:45:35.775036 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:35.774725 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a878660f-q4pkx" podStartSLOduration=2.774705268 podStartE2EDuration="2.774705268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:35.709299336 +0000 UTC m=+1911.263999118" watchObservedRunningTime="2026-10-30 21:45:35.774705268 +0000 UTC m=+1911.329405044" Oct 30 21:45:35.783515 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:35.783491 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40a39e54-dpfcc"] Oct 30 21:45:35.784022 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:35.784008 4560 cpu_manager.go:414] "containerMap: removing container" podUID="567f946c-fb5c-4798-b11c-bae6723f3d89" containerName="revoke-keys" Oct 30 21:45:35.784091 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:35.784024 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="567f946c-fb5c-4798-b11c-bae6723f3d89" containerName="revoke-keys" Oct 30 21:45:35.784137 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:35.784125 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="567f946c-fb5c-4798-b11c-bae6723f3d89" containerName="revoke-keys" Oct 30 21:45:35.787703 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:35.787685 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a39e54-dpfcc" Oct 30 21:45:35.824184 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:35.824162 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cf0de4-b9pvm" Oct 30 21:45:35.887918 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:35.887881 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40a39e54-dpfcc"] Oct 30 21:45:35.892831 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:35.892804 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7aaa2a11-2803-4d9d-9bf2-75175375cb8f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aaa2a11-2803-4d9d-9bf2-75175375cb8f-maas-api-xks-ca\") pod \"7aaa2a11-2803-4d9d-9bf2-75175375cb8f\" (UID: \"7aaa2a11-2803-4d9d-9bf2-75175375cb8f\") " Oct 30 21:45:35.892976 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:35.892884 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aaa2a11-2803-4d9d-9bf2-75175375cb8f-kube-api-access-2cpnr\" (UniqueName: \"kubernetes.io/projected/7aaa2a11-2803-4d9d-9bf2-75175375cb8f-kube-api-access-2cpnr\") pod \"7aaa2a11-2803-4d9d-9bf2-75175375cb8f\" (UID: \"7aaa2a11-2803-4d9d-9bf2-75175375cb8f\") " Oct 30 21:45:35.892976 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:35.892930 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aaa2a11-2803-4d9d-9bf2-75175375cb8f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aaa2a11-2803-4d9d-9bf2-75175375cb8f-maas-api-openshift-ca\") pod \"7aaa2a11-2803-4d9d-9bf2-75175375cb8f\" (UID: \"7aaa2a11-2803-4d9d-9bf2-75175375cb8f\") " Oct 30 21:45:35.893302 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:35.893277 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/223b6566-cc1e-49f4-866f-d8e09df28cd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40a39e54-dpfcc\" (UID: \"223b6566-cc1e-49f4-866f-d8e09df28cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a39e54-dpfcc" Oct 30 21:45:35.893478 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:35.893332 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/223b6566-cc1e-49f4-866f-d8e09df28cd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40a39e54-dpfcc\" (UID: \"223b6566-cc1e-49f4-866f-d8e09df28cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a39e54-dpfcc" Oct 30 21:45:35.893608 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:35.893586 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aaa2a11-2803-4d9d-9bf2-75175375cb8f-maas-api-openshift-ca" pod "7aaa2a11-2803-4d9d-9bf2-75175375cb8f" (UID: "7aaa2a11-2803-4d9d-9bf2-75175375cb8f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:35.893706 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:35.893684 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdj6l\" (UniqueName: \"kubernetes.io/projected/223b6566-cc1e-49f4-866f-d8e09df28cd9-kube-api-access-wdj6l\") pod \"maas-api-revoke-sub-40a39e54-dpfcc\" (UID: \"223b6566-cc1e-49f4-866f-d8e09df28cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a39e54-dpfcc" Oct 30 21:45:35.893816 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:35.893800 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aaa2a11-2803-4d9d-9bf2-75175375cb8f-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:35.898193 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:35.897182 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aaa2a11-2803-4d9d-9bf2-75175375cb8f-maas-api-xks-ca" pod "7aaa2a11-2803-4d9d-9bf2-75175375cb8f" (UID: "7aaa2a11-2803-4d9d-9bf2-75175375cb8f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:35.898193 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:35.898536 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aaa2a11-2803-4d9d-9bf2-75175375cb8f-kube-api-access-2cpnr" pod "7aaa2a11-2803-4d9d-9bf2-75175375cb8f" (UID: "7aaa2a11-2803-4d9d-9bf2-75175375cb8f"). InnerVolumeSpecName "kube-api-access-2cpnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:35.994557 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:35.994523 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/223b6566-cc1e-49f4-866f-d8e09df28cd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40a39e54-dpfcc\" (UID: \"223b6566-cc1e-49f4-866f-d8e09df28cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a39e54-dpfcc" Oct 30 21:45:35.994726 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:35.994568 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/223b6566-cc1e-49f4-866f-d8e09df28cd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40a39e54-dpfcc\" (UID: \"223b6566-cc1e-49f4-866f-d8e09df28cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a39e54-dpfcc" Oct 30 21:45:35.994726 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:35.994698 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdj6l\" (UniqueName: \"kubernetes.io/projected/223b6566-cc1e-49f4-866f-d8e09df28cd9-kube-api-access-wdj6l\") pod \"maas-api-revoke-sub-40a39e54-dpfcc\" (UID: \"223b6566-cc1e-49f4-866f-d8e09df28cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a39e54-dpfcc" Oct 30 21:45:35.994883 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:35.994867 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aaa2a11-2803-4d9d-9bf2-75175375cb8f-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:35.994918 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:35.994890 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cpnr\" (UniqueName: \"kubernetes.io/projected/7aaa2a11-2803-4d9d-9bf2-75175375cb8f-kube-api-access-2cpnr\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:35.995290 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:35.995272 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/223b6566-cc1e-49f4-866f-d8e09df28cd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40a39e54-dpfcc\" (UID: \"223b6566-cc1e-49f4-866f-d8e09df28cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a39e54-dpfcc" Oct 30 21:45:35.997117 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:35.997092 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/223b6566-cc1e-49f4-866f-d8e09df28cd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40a39e54-dpfcc\" (UID: \"223b6566-cc1e-49f4-866f-d8e09df28cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a39e54-dpfcc" Oct 30 21:45:36.057362 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:36.054860 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdj6l\" (UniqueName: \"kubernetes.io/projected/223b6566-cc1e-49f4-866f-d8e09df28cd9-kube-api-access-wdj6l\") pod \"maas-api-revoke-sub-40a39e54-dpfcc\" (UID: \"223b6566-cc1e-49f4-866f-d8e09df28cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a39e54-dpfcc" Oct 30 21:45:36.179610 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:36.179535 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a39e54-dpfcc" Oct 30 21:45:36.217617 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:36.217589 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ea0c294-8l2dw"] Oct 30 21:45:36.218122 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:36.218110 4560 cpu_manager.go:414] "containerMap: removing container" podUID="7aaa2a11-2803-4d9d-9bf2-75175375cb8f" containerName="revoke-keys" Oct 30 21:45:36.218169 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:36.218123 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aaa2a11-2803-4d9d-9bf2-75175375cb8f" containerName="revoke-keys" Oct 30 21:45:36.218209 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:36.218200 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aaa2a11-2803-4d9d-9bf2-75175375cb8f" containerName="revoke-keys" Oct 30 21:45:36.225563 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:36.225539 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea0c294-8l2dw" Oct 30 21:45:36.280496 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:36.280469 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ea0c294-8l2dw"] Oct 30 21:45:36.299185 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:36.299160 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d196a58-9be7-41e1-93cc-2348c2f7c30c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ea0c294-8l2dw\" (UID: \"6d196a58-9be7-41e1-93cc-2348c2f7c30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea0c294-8l2dw" Oct 30 21:45:36.299487 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:36.299448 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d196a58-9be7-41e1-93cc-2348c2f7c30c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ea0c294-8l2dw\" (UID: \"6d196a58-9be7-41e1-93cc-2348c2f7c30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea0c294-8l2dw" Oct 30 21:45:36.299580 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:36.299503 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw2tr\" (UniqueName: \"kubernetes.io/projected/6d196a58-9be7-41e1-93cc-2348c2f7c30c-kube-api-access-fw2tr\") pod \"maas-api-revoke-sub-2ea0c294-8l2dw\" (UID: \"6d196a58-9be7-41e1-93cc-2348c2f7c30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea0c294-8l2dw" Oct 30 21:45:36.401173 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:36.401138 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d196a58-9be7-41e1-93cc-2348c2f7c30c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ea0c294-8l2dw\" (UID: \"6d196a58-9be7-41e1-93cc-2348c2f7c30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea0c294-8l2dw" Oct 30 21:45:36.401365 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:36.401287 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d196a58-9be7-41e1-93cc-2348c2f7c30c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ea0c294-8l2dw\" (UID: \"6d196a58-9be7-41e1-93cc-2348c2f7c30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea0c294-8l2dw" Oct 30 21:45:36.401365 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:36.401307 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw2tr\" (UniqueName: \"kubernetes.io/projected/6d196a58-9be7-41e1-93cc-2348c2f7c30c-kube-api-access-fw2tr\") pod \"maas-api-revoke-sub-2ea0c294-8l2dw\" (UID: \"6d196a58-9be7-41e1-93cc-2348c2f7c30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea0c294-8l2dw" Oct 30 21:45:36.401975 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:36.401947 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d196a58-9be7-41e1-93cc-2348c2f7c30c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ea0c294-8l2dw\" (UID: \"6d196a58-9be7-41e1-93cc-2348c2f7c30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea0c294-8l2dw" Oct 30 21:45:36.403598 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:36.403577 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d196a58-9be7-41e1-93cc-2348c2f7c30c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ea0c294-8l2dw\" (UID: \"6d196a58-9be7-41e1-93cc-2348c2f7c30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea0c294-8l2dw" Oct 30 21:45:36.460851 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:36.460783 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw2tr\" (UniqueName: \"kubernetes.io/projected/6d196a58-9be7-41e1-93cc-2348c2f7c30c-kube-api-access-fw2tr\") pod \"maas-api-revoke-sub-2ea0c294-8l2dw\" (UID: \"6d196a58-9be7-41e1-93cc-2348c2f7c30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea0c294-8l2dw" Oct 30 21:45:36.576080 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:36.574649 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea0c294-8l2dw" Oct 30 21:45:36.589432 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:36.589404 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cf0de4-b9pvm" Oct 30 21:45:36.589854 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:36.589443 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cf0de4-b9pvm" event={"ID":"7aaa2a11-2803-4d9d-9bf2-75175375cb8f","Type":"ContainerDied","Data":"54a152f6c67f19aebc4d98702d20e933f35dc52531fe23bae4a703fb8ecb36ab"} Oct 30 21:45:36.590029 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:36.590003 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54a152f6c67f19aebc4d98702d20e933f35dc52531fe23bae4a703fb8ecb36ab" Oct 30 21:45:36.592412 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:36.591277 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40a39e54-dpfcc"] Oct 30 21:45:36.592412 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:36.591736 4560 generic.go:356] "Generic (PLEG): container finished" podID="ace52bf5-0ea8-4d12-94b9-e234c93f8720" containerID="7bb4dc5ee6d7c667bb8b520957a0c46fb0be949333a9ad9ce2db6fee1b83872a" exitCode=0 Oct 30 21:45:36.592412 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:36.591809 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a878660f-q4pkx" event={"ID":"ace52bf5-0ea8-4d12-94b9-e234c93f8720","Type":"ContainerDied","Data":"7bb4dc5ee6d7c667bb8b520957a0c46fb0be949333a9ad9ce2db6fee1b83872a"} Oct 30 21:45:36.754247 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:36.754213 4560 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 21:45:36.754530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:36.754499 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" podUID="57d2540b-40a0-4326-a4af-57466f9abbae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 21:45:36.989642 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:36.989612 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ea0c294-8l2dw"] Oct 30 21:45:36.993126 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:45:36.993100 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d196a58_9be7_41e1_93cc_2348c2f7c30c.slice/crio-7807feab6adc4c601673c1ddac843fcfb50372fb45e7125e846a7af0b9acdde5 WatchSource:0}: Error finding container 7807feab6adc4c601673c1ddac843fcfb50372fb45e7125e846a7af0b9acdde5: Status 404 returned error can't find the container with id 7807feab6adc4c601673c1ddac843fcfb50372fb45e7125e846a7af0b9acdde5 Oct 30 21:45:37.091427 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:37.091391 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84fa9d95-csxcd"] Oct 30 21:45:37.100589 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:37.095668 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84fa9d95-csxcd"] Oct 30 21:45:37.601317 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:37.601139 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea0c294-8l2dw" event={"ID":"6d196a58-9be7-41e1-93cc-2348c2f7c30c","Type":"ContainerStarted","Data":"8183db8753050c39913c87ff7768213c7edd7587a91695e0928fc601351ea1be"} Oct 30 21:45:37.601317 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:37.601205 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea0c294-8l2dw" event={"ID":"6d196a58-9be7-41e1-93cc-2348c2f7c30c","Type":"ContainerStarted","Data":"7807feab6adc4c601673c1ddac843fcfb50372fb45e7125e846a7af0b9acdde5"} Oct 30 21:45:37.603389 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:37.603359 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a39e54-dpfcc" event={"ID":"223b6566-cc1e-49f4-866f-d8e09df28cd9","Type":"ContainerStarted","Data":"0e4ad9b07bec93e19a8b3c854212292836d678d0b6494538611bf1aae2b46ff6"} Oct 30 21:45:37.603528 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:37.603405 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a39e54-dpfcc" event={"ID":"223b6566-cc1e-49f4-866f-d8e09df28cd9","Type":"ContainerStarted","Data":"c651da256e83df6cc0c872b3d49db2f03be9e72411fd5fd46b3748257ff73345"} Oct 30 21:45:37.717061 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:37.716956 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea0c294-8l2dw" podStartSLOduration=1.716936657 podStartE2EDuration="1.716936657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:37.715234111 +0000 UTC m=+1913.269933888" watchObservedRunningTime="2026-10-30 21:45:37.716936657 +0000 UTC m=+1913.271636437" Oct 30 21:45:37.757380 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:37.740561 4560 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 21:45:37.757380 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:37.740622 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" podUID="57d2540b-40a0-4326-a4af-57466f9abbae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 21:45:37.862889 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:37.862841 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a39e54-dpfcc" podStartSLOduration=2.862825405 podStartE2EDuration="2.862825405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:37.859855489 +0000 UTC m=+1913.414555266" watchObservedRunningTime="2026-10-30 21:45:37.862825405 +0000 UTC m=+1913.417525181" Oct 30 21:45:37.914581 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:37.914556 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a878660f-q4pkx" Oct 30 21:45:38.063006 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:38.062976 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ace52bf5-0ea8-4d12-94b9-e234c93f8720-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ace52bf5-0ea8-4d12-94b9-e234c93f8720-maas-api-xks-ca\") pod \"ace52bf5-0ea8-4d12-94b9-e234c93f8720\" (UID: \"ace52bf5-0ea8-4d12-94b9-e234c93f8720\") " Oct 30 21:45:38.063176 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:38.063105 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ace52bf5-0ea8-4d12-94b9-e234c93f8720-kube-api-access-w6cmf\" (UniqueName: \"kubernetes.io/projected/ace52bf5-0ea8-4d12-94b9-e234c93f8720-kube-api-access-w6cmf\") pod \"ace52bf5-0ea8-4d12-94b9-e234c93f8720\" (UID: \"ace52bf5-0ea8-4d12-94b9-e234c93f8720\") " Oct 30 21:45:38.063315 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:38.063247 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ace52bf5-0ea8-4d12-94b9-e234c93f8720-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ace52bf5-0ea8-4d12-94b9-e234c93f8720-maas-api-openshift-ca\") pod \"ace52bf5-0ea8-4d12-94b9-e234c93f8720\" (UID: \"ace52bf5-0ea8-4d12-94b9-e234c93f8720\") " Oct 30 21:45:38.064092 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:38.064065 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ace52bf5-0ea8-4d12-94b9-e234c93f8720-maas-api-openshift-ca" pod "ace52bf5-0ea8-4d12-94b9-e234c93f8720" (UID: "ace52bf5-0ea8-4d12-94b9-e234c93f8720"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:38.076911 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:38.076885 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5d8ac1a-6jgw5"] Oct 30 21:45:38.080979 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:38.080924 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ace52bf5-0ea8-4d12-94b9-e234c93f8720-maas-api-xks-ca" pod "ace52bf5-0ea8-4d12-94b9-e234c93f8720" (UID: "ace52bf5-0ea8-4d12-94b9-e234c93f8720"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:38.083500 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:38.083473 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ace52bf5-0ea8-4d12-94b9-e234c93f8720-kube-api-access-w6cmf" pod "ace52bf5-0ea8-4d12-94b9-e234c93f8720" (UID: "ace52bf5-0ea8-4d12-94b9-e234c93f8720"). InnerVolumeSpecName "kube-api-access-w6cmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:38.083950 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:38.083933 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5d8ac1a-6jgw5"] Oct 30 21:45:38.165610 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:38.165582 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ace52bf5-0ea8-4d12-94b9-e234c93f8720-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:38.165610 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:38.165610 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6cmf\" (UniqueName: \"kubernetes.io/projected/ace52bf5-0ea8-4d12-94b9-e234c93f8720-kube-api-access-w6cmf\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:38.165770 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:38.165620 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ace52bf5-0ea8-4d12-94b9-e234c93f8720-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:38.577801 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:38.577762 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a42988c9-a7e6-4dde-88d9-73ea4586dd79" path="/var/lib/kubelet/pods/a42988c9-a7e6-4dde-88d9-73ea4586dd79/volumes" Oct 30 21:45:38.578391 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:38.578373 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba20dae7-d119-48cb-9f68-f8619ebd303b" path="/var/lib/kubelet/pods/ba20dae7-d119-48cb-9f68-f8619ebd303b/volumes" Oct 30 21:45:38.611856 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:38.611829 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a878660f-q4pkx" Oct 30 21:45:38.611856 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:38.611834 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a878660f-q4pkx" event={"ID":"ace52bf5-0ea8-4d12-94b9-e234c93f8720","Type":"ContainerDied","Data":"36f629ad6eb41c155557b8bd95c39eeff63f4a716dbbcbe20d6c1e8d89f6df5d"} Oct 30 21:45:38.612333 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:38.611893 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36f629ad6eb41c155557b8bd95c39eeff63f4a716dbbcbe20d6c1e8d89f6df5d" Oct 30 21:45:38.613630 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:38.613604 4560 generic.go:356] "Generic (PLEG): container finished" podID="6d196a58-9be7-41e1-93cc-2348c2f7c30c" containerID="8183db8753050c39913c87ff7768213c7edd7587a91695e0928fc601351ea1be" exitCode=0 Oct 30 21:45:38.613838 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:38.613646 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea0c294-8l2dw" event={"ID":"6d196a58-9be7-41e1-93cc-2348c2f7c30c","Type":"ContainerDied","Data":"8183db8753050c39913c87ff7768213c7edd7587a91695e0928fc601351ea1be"} Oct 30 21:45:38.615577 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:38.615552 4560 generic.go:356] "Generic (PLEG): container finished" podID="223b6566-cc1e-49f4-866f-d8e09df28cd9" containerID="0e4ad9b07bec93e19a8b3c854212292836d678d0b6494538611bf1aae2b46ff6" exitCode=0 Oct 30 21:45:38.615685 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:38.615635 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a39e54-dpfcc" event={"ID":"223b6566-cc1e-49f4-866f-d8e09df28cd9","Type":"ContainerDied","Data":"0e4ad9b07bec93e19a8b3c854212292836d678d0b6494538611bf1aae2b46ff6"} Oct 30 21:45:38.746361 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:38.731710 4560 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 21:45:38.746361 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:38.731768 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" podUID="57d2540b-40a0-4326-a4af-57466f9abbae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 21:45:39.730369 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:39.728723 4560 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 21:45:39.730369 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:39.728783 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" podUID="57d2540b-40a0-4326-a4af-57466f9abbae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 21:45:39.788869 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:39.788836 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea0c294-8l2dw" Oct 30 21:45:39.835603 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:39.835575 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a39e54-dpfcc" Oct 30 21:45:39.888724 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:39.888646 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d196a58-9be7-41e1-93cc-2348c2f7c30c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d196a58-9be7-41e1-93cc-2348c2f7c30c-maas-api-openshift-ca\") pod \"6d196a58-9be7-41e1-93cc-2348c2f7c30c\" (UID: \"6d196a58-9be7-41e1-93cc-2348c2f7c30c\") " Oct 30 21:45:39.888876 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:39.888760 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d196a58-9be7-41e1-93cc-2348c2f7c30c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d196a58-9be7-41e1-93cc-2348c2f7c30c-maas-api-xks-ca\") pod \"6d196a58-9be7-41e1-93cc-2348c2f7c30c\" (UID: \"6d196a58-9be7-41e1-93cc-2348c2f7c30c\") " Oct 30 21:45:39.888960 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:39.888875 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d196a58-9be7-41e1-93cc-2348c2f7c30c-kube-api-access-fw2tr\" (UniqueName: \"kubernetes.io/projected/6d196a58-9be7-41e1-93cc-2348c2f7c30c-kube-api-access-fw2tr\") pod \"6d196a58-9be7-41e1-93cc-2348c2f7c30c\" (UID: \"6d196a58-9be7-41e1-93cc-2348c2f7c30c\") " Oct 30 21:45:39.889021 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:39.888985 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d196a58-9be7-41e1-93cc-2348c2f7c30c-maas-api-openshift-ca" pod "6d196a58-9be7-41e1-93cc-2348c2f7c30c" (UID: "6d196a58-9be7-41e1-93cc-2348c2f7c30c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:39.889495 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:39.889473 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d196a58-9be7-41e1-93cc-2348c2f7c30c-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:39.890915 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:39.890887 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d196a58-9be7-41e1-93cc-2348c2f7c30c-kube-api-access-fw2tr" pod "6d196a58-9be7-41e1-93cc-2348c2f7c30c" (UID: "6d196a58-9be7-41e1-93cc-2348c2f7c30c"). InnerVolumeSpecName "kube-api-access-fw2tr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:39.890990 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:39.890964 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d196a58-9be7-41e1-93cc-2348c2f7c30c-maas-api-xks-ca" pod "6d196a58-9be7-41e1-93cc-2348c2f7c30c" (UID: "6d196a58-9be7-41e1-93cc-2348c2f7c30c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:39.990711 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:39.990681 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/223b6566-cc1e-49f4-866f-d8e09df28cd9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/223b6566-cc1e-49f4-866f-d8e09df28cd9-maas-api-xks-ca\") pod \"223b6566-cc1e-49f4-866f-d8e09df28cd9\" (UID: \"223b6566-cc1e-49f4-866f-d8e09df28cd9\") " Oct 30 21:45:39.990912 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:39.990745 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/223b6566-cc1e-49f4-866f-d8e09df28cd9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/223b6566-cc1e-49f4-866f-d8e09df28cd9-maas-api-openshift-ca\") pod \"223b6566-cc1e-49f4-866f-d8e09df28cd9\" (UID: \"223b6566-cc1e-49f4-866f-d8e09df28cd9\") " Oct 30 21:45:39.990912 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:39.990805 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/223b6566-cc1e-49f4-866f-d8e09df28cd9-kube-api-access-wdj6l\" (UniqueName: \"kubernetes.io/projected/223b6566-cc1e-49f4-866f-d8e09df28cd9-kube-api-access-wdj6l\") pod \"223b6566-cc1e-49f4-866f-d8e09df28cd9\" (UID: \"223b6566-cc1e-49f4-866f-d8e09df28cd9\") " Oct 30 21:45:39.991094 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:39.991069 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/223b6566-cc1e-49f4-866f-d8e09df28cd9-maas-api-openshift-ca" pod "223b6566-cc1e-49f4-866f-d8e09df28cd9" (UID: "223b6566-cc1e-49f4-866f-d8e09df28cd9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:39.991502 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:39.991479 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d196a58-9be7-41e1-93cc-2348c2f7c30c-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:39.991502 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:39.991501 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw2tr\" (UniqueName: \"kubernetes.io/projected/6d196a58-9be7-41e1-93cc-2348c2f7c30c-kube-api-access-fw2tr\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:39.991662 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:39.991510 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/223b6566-cc1e-49f4-866f-d8e09df28cd9-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:39.992807 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:39.992787 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/223b6566-cc1e-49f4-866f-d8e09df28cd9-kube-api-access-wdj6l" pod "223b6566-cc1e-49f4-866f-d8e09df28cd9" (UID: "223b6566-cc1e-49f4-866f-d8e09df28cd9"). InnerVolumeSpecName "kube-api-access-wdj6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:39.992923 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:39.992834 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/223b6566-cc1e-49f4-866f-d8e09df28cd9-maas-api-xks-ca" pod "223b6566-cc1e-49f4-866f-d8e09df28cd9" (UID: "223b6566-cc1e-49f4-866f-d8e09df28cd9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:40.092427 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:40.092390 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/223b6566-cc1e-49f4-866f-d8e09df28cd9-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:40.092427 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:40.092416 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdj6l\" (UniqueName: \"kubernetes.io/projected/223b6566-cc1e-49f4-866f-d8e09df28cd9-kube-api-access-wdj6l\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:40.631002 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:40.630966 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea0c294-8l2dw" event={"ID":"6d196a58-9be7-41e1-93cc-2348c2f7c30c","Type":"ContainerDied","Data":"7807feab6adc4c601673c1ddac843fcfb50372fb45e7125e846a7af0b9acdde5"} Oct 30 21:45:40.631124 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:40.631024 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7807feab6adc4c601673c1ddac843fcfb50372fb45e7125e846a7af0b9acdde5" Oct 30 21:45:40.631124 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:40.630975 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea0c294-8l2dw" Oct 30 21:45:40.632477 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:40.632452 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a39e54-dpfcc" Oct 30 21:45:40.632587 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:40.632495 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a39e54-dpfcc" event={"ID":"223b6566-cc1e-49f4-866f-d8e09df28cd9","Type":"ContainerDied","Data":"c651da256e83df6cc0c872b3d49db2f03be9e72411fd5fd46b3748257ff73345"} Oct 30 21:45:40.632587 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:40.632542 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c651da256e83df6cc0c872b3d49db2f03be9e72411fd5fd46b3748257ff73345" Oct 30 21:45:40.732891 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:40.732862 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" Oct 30 21:45:40.733225 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:40.733104 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" Oct 30 21:45:40.733935 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:40.733915 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" Oct 30 21:45:41.049192 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:41.049162 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2092c607-pw5m4"] Oct 30 21:45:41.053015 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:41.052985 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2092c607-pw5m4"] Oct 30 21:45:42.576950 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:42.576917 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f58aea94-9df8-4efc-9ce3-bcaddf98826c" path="/var/lib/kubelet/pods/f58aea94-9df8-4efc-9ce3-bcaddf98826c/volumes" Oct 30 21:45:44.948938 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:44.948911 4560 scope.go:122] "RemoveContainer" containerID="11e82a0aa5b0e62b53cc9637f57c0b826bcbd1d0d344fbefea56084fc882295e" Oct 30 21:45:44.977392 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:44.977324 4560 scope.go:122] "RemoveContainer" containerID="0879e2e115dd38598bc8e53662342d20ac703890bcc95945b362c9ac1aa41667" Oct 30 21:45:44.987327 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:44.987306 4560 scope.go:122] "RemoveContainer" containerID="8dc0a87edb830e988a276c29eec9b058dbc7127ef572fcc38dab4622b2bb801b" Oct 30 21:45:44.995774 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:44.995738 4560 scope.go:122] "RemoveContainer" containerID="37b3f9c7df364cd509d38459eea6c04f135ff32c7c543c6203c3d53be99e4b7b" Oct 30 21:45:45.006744 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:45.006723 4560 scope.go:122] "RemoveContainer" containerID="b6ef102f57beccaec3966804bfb7d420ed218f53efa8dc4f2a8111627bf0748c" Oct 30 21:45:45.052809 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:45.052786 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df14ffda-gmsd2"] Oct 30 21:45:45.098930 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:45.098906 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df14ffda-gmsd2"] Oct 30 21:45:45.759442 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:45.759405 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-910e9fa6-ghrtx"] Oct 30 21:45:45.759911 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:45.759897 4560 cpu_manager.go:414] "containerMap: removing container" podUID="6d196a58-9be7-41e1-93cc-2348c2f7c30c" containerName="revoke-keys" Oct 30 21:45:45.759982 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:45.759914 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d196a58-9be7-41e1-93cc-2348c2f7c30c" containerName="revoke-keys" Oct 30 21:45:45.759982 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:45.759930 4560 cpu_manager.go:414] "containerMap: removing container" podUID="223b6566-cc1e-49f4-866f-d8e09df28cd9" containerName="revoke-keys" Oct 30 21:45:45.759982 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:45.759935 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="223b6566-cc1e-49f4-866f-d8e09df28cd9" containerName="revoke-keys" Oct 30 21:45:45.759982 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:45.759955 4560 cpu_manager.go:414] "containerMap: removing container" podUID="ace52bf5-0ea8-4d12-94b9-e234c93f8720" containerName="revoke-keys" Oct 30 21:45:45.759982 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:45.759960 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ace52bf5-0ea8-4d12-94b9-e234c93f8720" containerName="revoke-keys" Oct 30 21:45:45.760230 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:45.760091 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="ace52bf5-0ea8-4d12-94b9-e234c93f8720" containerName="revoke-keys" Oct 30 21:45:45.760230 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:45.760109 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d196a58-9be7-41e1-93cc-2348c2f7c30c" containerName="revoke-keys" Oct 30 21:45:45.760230 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:45.760121 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="223b6566-cc1e-49f4-866f-d8e09df28cd9" containerName="revoke-keys" Oct 30 21:45:45.765510 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:45.765492 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-910e9fa6-ghrtx" Oct 30 21:45:45.820283 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:45.820258 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-910e9fa6-ghrtx"] Oct 30 21:45:45.861455 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:45.861431 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:45:45.861455 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:45.861446 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:45:45.861618 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:45.861537 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:45:45.861618 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:45.861579 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:45:45.962924 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:45.962895 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfwwx\" (UniqueName: \"kubernetes.io/projected/e109d452-56a4-47f8-bbd6-cea3421ad8c7-kube-api-access-jfwwx\") pod \"maas-api-revoke-sub-910e9fa6-ghrtx\" (UID: \"e109d452-56a4-47f8-bbd6-cea3421ad8c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-910e9fa6-ghrtx" Oct 30 21:45:45.963370 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:45.962938 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e109d452-56a4-47f8-bbd6-cea3421ad8c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-910e9fa6-ghrtx\" (UID: \"e109d452-56a4-47f8-bbd6-cea3421ad8c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-910e9fa6-ghrtx" Oct 30 21:45:45.963370 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:45.963078 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e109d452-56a4-47f8-bbd6-cea3421ad8c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-910e9fa6-ghrtx\" (UID: \"e109d452-56a4-47f8-bbd6-cea3421ad8c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-910e9fa6-ghrtx" Oct 30 21:45:46.063816 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:46.063719 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e109d452-56a4-47f8-bbd6-cea3421ad8c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-910e9fa6-ghrtx\" (UID: \"e109d452-56a4-47f8-bbd6-cea3421ad8c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-910e9fa6-ghrtx" Oct 30 21:45:46.063935 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:46.063889 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfwwx\" (UniqueName: \"kubernetes.io/projected/e109d452-56a4-47f8-bbd6-cea3421ad8c7-kube-api-access-jfwwx\") pod \"maas-api-revoke-sub-910e9fa6-ghrtx\" (UID: \"e109d452-56a4-47f8-bbd6-cea3421ad8c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-910e9fa6-ghrtx" Oct 30 21:45:46.063935 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:46.063927 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e109d452-56a4-47f8-bbd6-cea3421ad8c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-910e9fa6-ghrtx\" (UID: \"e109d452-56a4-47f8-bbd6-cea3421ad8c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-910e9fa6-ghrtx" Oct 30 21:45:46.064533 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:46.064514 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e109d452-56a4-47f8-bbd6-cea3421ad8c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-910e9fa6-ghrtx\" (UID: \"e109d452-56a4-47f8-bbd6-cea3421ad8c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-910e9fa6-ghrtx" Oct 30 21:45:46.066236 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:46.066215 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e109d452-56a4-47f8-bbd6-cea3421ad8c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-910e9fa6-ghrtx\" (UID: \"e109d452-56a4-47f8-bbd6-cea3421ad8c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-910e9fa6-ghrtx" Oct 30 21:45:46.122246 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:46.122223 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfwwx\" (UniqueName: \"kubernetes.io/projected/e109d452-56a4-47f8-bbd6-cea3421ad8c7-kube-api-access-jfwwx\") pod \"maas-api-revoke-sub-910e9fa6-ghrtx\" (UID: \"e109d452-56a4-47f8-bbd6-cea3421ad8c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-910e9fa6-ghrtx" Oct 30 21:45:46.378908 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:46.378834 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-910e9fa6-ghrtx" Oct 30 21:45:46.577046 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:46.577008 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfba7e24-87b2-4396-991f-4f38aed49369" path="/var/lib/kubelet/pods/bfba7e24-87b2-4396-991f-4f38aed49369/volumes" Oct 30 21:45:46.779258 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:46.779221 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-910e9fa6-ghrtx"] Oct 30 21:45:46.783099 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:45:46.783075 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode109d452_56a4_47f8_bbd6_cea3421ad8c7.slice/crio-5966cf6143c74871c0d7a0b648d6b59e99d40834fbaf4d6c6aab7a4ab7162c2f WatchSource:0}: Error finding container 5966cf6143c74871c0d7a0b648d6b59e99d40834fbaf4d6c6aab7a4ab7162c2f: Status 404 returned error can't find the container with id 5966cf6143c74871c0d7a0b648d6b59e99d40834fbaf4d6c6aab7a4ab7162c2f Oct 30 21:45:47.707842 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:47.707808 4560 generic.go:356] "Generic (PLEG): container finished" podID="e109d452-56a4-47f8-bbd6-cea3421ad8c7" containerID="34c6762cd084909ae52bccaf07e4872089b260b9072ac34ed0b9f9634e12afc4" exitCode=0 Oct 30 21:45:47.708203 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:47.707864 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-910e9fa6-ghrtx" event={"ID":"e109d452-56a4-47f8-bbd6-cea3421ad8c7","Type":"ContainerDied","Data":"34c6762cd084909ae52bccaf07e4872089b260b9072ac34ed0b9f9634e12afc4"} Oct 30 21:45:47.708203 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:47.707922 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-910e9fa6-ghrtx" event={"ID":"e109d452-56a4-47f8-bbd6-cea3421ad8c7","Type":"ContainerStarted","Data":"5966cf6143c74871c0d7a0b648d6b59e99d40834fbaf4d6c6aab7a4ab7162c2f"} Oct 30 21:45:48.530734 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:48.530701 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8"] Oct 30 21:45:48.534877 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:48.534855 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" Oct 30 21:45:48.596621 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:48.596596 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-istio-data\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8\" (UID: \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" Oct 30 21:45:48.596769 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:48.596748 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-workload-socket\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8\" (UID: \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" Oct 30 21:45:48.596850 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:48.596833 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-credential-socket\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8\" (UID: \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" Oct 30 21:45:48.596890 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:48.596866 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkcgd\" (UniqueName: \"kubernetes.io/projected/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-kube-api-access-hkcgd\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8\" (UID: \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" Oct 30 21:45:48.596927 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:48.596888 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-istiod-ca-cert\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8\" (UID: \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" Oct 30 21:45:48.596927 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:48.596912 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-istio-token\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8\" (UID: \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" Oct 30 21:45:48.597079 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:48.597059 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-workload-certs\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8\" (UID: \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" Oct 30 21:45:48.597117 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:48.597097 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-istio-envoy\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8\" (UID: \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" Oct 30 21:45:48.597165 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:48.597149 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-istio-podinfo\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8\" (UID: \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" Oct 30 21:45:48.630200 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:48.630174 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8"] Oct 30 21:45:48.698610 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:48.698583 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-istio-podinfo\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8\" (UID: \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" Oct 30 21:45:48.698771 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:48.698639 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-istio-data\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8\" (UID: \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" Oct 30 21:45:48.698771 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:48.698729 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-workload-socket\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8\" (UID: \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" Oct 30 21:45:48.698860 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:48.698805 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-credential-socket\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8\" (UID: \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" Oct 30 21:45:48.698860 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:48.698841 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkcgd\" (UniqueName: \"kubernetes.io/projected/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-kube-api-access-hkcgd\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8\" (UID: \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" Oct 30 21:45:48.698952 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:48.698870 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-istiod-ca-cert\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8\" (UID: \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" Oct 30 21:45:48.698952 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:48.698922 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-istio-token\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8\" (UID: \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" Oct 30 21:45:48.699055 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:48.698988 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-workload-certs\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8\" (UID: \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" Oct 30 21:45:48.699107 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:48.699070 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-istio-data\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8\" (UID: \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" Oct 30 21:45:48.699161 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:48.699119 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-workload-socket\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8\" (UID: \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" Oct 30 21:45:48.699233 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:48.699193 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-istio-envoy\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8\" (UID: \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" Oct 30 21:45:48.699474 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:48.699447 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-credential-socket\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8\" (UID: \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" Oct 30 21:45:48.699558 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:48.699529 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-workload-certs\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8\" (UID: \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" Oct 30 21:45:48.700372 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:48.700306 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-istiod-ca-cert\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8\" (UID: \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" Oct 30 21:45:48.701902 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:48.701862 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-istio-podinfo\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8\" (UID: \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" Oct 30 21:45:48.702003 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:48.701913 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-istio-envoy\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8\" (UID: \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" Oct 30 21:45:48.757179 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:48.757141 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-istio-token\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8\" (UID: \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" Oct 30 21:45:48.757562 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:48.757461 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkcgd\" (UniqueName: \"kubernetes.io/projected/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-kube-api-access-hkcgd\") pod \"e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8\" (UID: \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\") " pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" Oct 30 21:45:48.848062 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:48.847949 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" Oct 30 21:45:48.883041 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:48.883020 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-910e9fa6-ghrtx" Oct 30 21:45:48.902268 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:48.902239 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e109d452-56a4-47f8-bbd6-cea3421ad8c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e109d452-56a4-47f8-bbd6-cea3421ad8c7-maas-api-xks-ca\") pod \"e109d452-56a4-47f8-bbd6-cea3421ad8c7\" (UID: \"e109d452-56a4-47f8-bbd6-cea3421ad8c7\") " Oct 30 21:45:48.902411 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:48.902286 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e109d452-56a4-47f8-bbd6-cea3421ad8c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e109d452-56a4-47f8-bbd6-cea3421ad8c7-maas-api-openshift-ca\") pod \"e109d452-56a4-47f8-bbd6-cea3421ad8c7\" (UID: \"e109d452-56a4-47f8-bbd6-cea3421ad8c7\") " Oct 30 21:45:48.902411 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:48.902350 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e109d452-56a4-47f8-bbd6-cea3421ad8c7-kube-api-access-jfwwx\" (UniqueName: \"kubernetes.io/projected/e109d452-56a4-47f8-bbd6-cea3421ad8c7-kube-api-access-jfwwx\") pod \"e109d452-56a4-47f8-bbd6-cea3421ad8c7\" (UID: \"e109d452-56a4-47f8-bbd6-cea3421ad8c7\") " Oct 30 21:45:48.903249 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:48.903226 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e109d452-56a4-47f8-bbd6-cea3421ad8c7-maas-api-openshift-ca" pod "e109d452-56a4-47f8-bbd6-cea3421ad8c7" (UID: "e109d452-56a4-47f8-bbd6-cea3421ad8c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:48.905006 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:48.904982 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e109d452-56a4-47f8-bbd6-cea3421ad8c7-maas-api-xks-ca" pod "e109d452-56a4-47f8-bbd6-cea3421ad8c7" (UID: "e109d452-56a4-47f8-bbd6-cea3421ad8c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:48.905236 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:48.905213 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e109d452-56a4-47f8-bbd6-cea3421ad8c7-kube-api-access-jfwwx" pod "e109d452-56a4-47f8-bbd6-cea3421ad8c7" (UID: "e109d452-56a4-47f8-bbd6-cea3421ad8c7"). InnerVolumeSpecName "kube-api-access-jfwwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:49.004584 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:49.004550 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e109d452-56a4-47f8-bbd6-cea3421ad8c7-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:49.004584 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:49.004578 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e109d452-56a4-47f8-bbd6-cea3421ad8c7-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:49.004584 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:49.004589 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfwwx\" (UniqueName: \"kubernetes.io/projected/e109d452-56a4-47f8-bbd6-cea3421ad8c7-kube-api-access-jfwwx\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:49.291652 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:49.291613 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8"] Oct 30 21:45:49.295503 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:45:49.295476 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ee3ad34_eadf_4009_ac4f_bcc9ee000ce4.slice/crio-21d32539cd13fce101fb82f51a83d71e1f9f3a03e035cd3d8261e59d8e907181 WatchSource:0}: Error finding container 21d32539cd13fce101fb82f51a83d71e1f9f3a03e035cd3d8261e59d8e907181: Status 404 returned error can't find the container with id 21d32539cd13fce101fb82f51a83d71e1f9f3a03e035cd3d8261e59d8e907181 Oct 30 21:45:49.298069 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:49.298038 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:45:49.298145 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:49.298103 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:45:49.298145 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:49.298132 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:45:49.687877 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:45:49.687793 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:45:49.726006 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:49.725971 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" event={"ID":"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4","Type":"ContainerStarted","Data":"303d0c48ce35ea81add41aed6efd79a2c281718a209cea2d4e5646bbbfd98264"} Oct 30 21:45:49.726150 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:49.726038 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" event={"ID":"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4","Type":"ContainerStarted","Data":"21d32539cd13fce101fb82f51a83d71e1f9f3a03e035cd3d8261e59d8e907181"} Oct 30 21:45:49.727491 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:49.727471 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-910e9fa6-ghrtx" Oct 30 21:45:49.727613 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:49.727506 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-910e9fa6-ghrtx" event={"ID":"e109d452-56a4-47f8-bbd6-cea3421ad8c7","Type":"ContainerDied","Data":"5966cf6143c74871c0d7a0b648d6b59e99d40834fbaf4d6c6aab7a4ab7162c2f"} Oct 30 21:45:49.727613 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:49.727545 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5966cf6143c74871c0d7a0b648d6b59e99d40834fbaf4d6c6aab7a4ab7162c2f" Oct 30 21:45:49.844362 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:49.843916 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" podStartSLOduration=1.843897032 podStartE2EDuration="1.843897032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:49.842255974 +0000 UTC m=+1925.396955807" watchObservedRunningTime="2026-10-30 21:45:49.843897032 +0000 UTC m=+1925.398596810" Oct 30 21:45:49.850372 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:49.849414 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" Oct 30 21:45:50.165170 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:50.165122 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf6e2-w9zhs"] Oct 30 21:45:50.165647 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:50.165632 4560 cpu_manager.go:414] "containerMap: removing container" podUID="e109d452-56a4-47f8-bbd6-cea3421ad8c7" containerName="revoke-keys" Oct 30 21:45:50.165647 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:50.165647 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e109d452-56a4-47f8-bbd6-cea3421ad8c7" containerName="revoke-keys" Oct 30 21:45:50.165775 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:50.165742 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="e109d452-56a4-47f8-bbd6-cea3421ad8c7" containerName="revoke-keys" Oct 30 21:45:50.169225 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:50.169207 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf6e2-w9zhs" Oct 30 21:45:50.221660 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:50.221626 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d69cebfb-2af0-4064-b06e-9a4e3a09d43b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1ccf6e2-w9zhs\" (UID: \"d69cebfb-2af0-4064-b06e-9a4e3a09d43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf6e2-w9zhs" Oct 30 21:45:50.221819 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:50.221760 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hz8c\" (UniqueName: \"kubernetes.io/projected/d69cebfb-2af0-4064-b06e-9a4e3a09d43b-kube-api-access-2hz8c\") pod \"maas-api-revoke-sub-a1ccf6e2-w9zhs\" (UID: \"d69cebfb-2af0-4064-b06e-9a4e3a09d43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf6e2-w9zhs" Oct 30 21:45:50.221876 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:50.221842 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d69cebfb-2af0-4064-b06e-9a4e3a09d43b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1ccf6e2-w9zhs\" (UID: \"d69cebfb-2af0-4064-b06e-9a4e3a09d43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf6e2-w9zhs" Oct 30 21:45:50.239361 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:50.230593 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:45:50.239361 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:50.230686 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:45:50.239361 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:50.230781 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:45:50.243356 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:50.240528 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:45:50.260390 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:50.259187 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf6e2-w9zhs"] Oct 30 21:45:50.323444 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:50.323413 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d69cebfb-2af0-4064-b06e-9a4e3a09d43b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1ccf6e2-w9zhs\" (UID: \"d69cebfb-2af0-4064-b06e-9a4e3a09d43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf6e2-w9zhs" Oct 30 21:45:50.323595 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:50.323488 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hz8c\" (UniqueName: \"kubernetes.io/projected/d69cebfb-2af0-4064-b06e-9a4e3a09d43b-kube-api-access-2hz8c\") pod \"maas-api-revoke-sub-a1ccf6e2-w9zhs\" (UID: \"d69cebfb-2af0-4064-b06e-9a4e3a09d43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf6e2-w9zhs" Oct 30 21:45:50.323595 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:50.323516 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d69cebfb-2af0-4064-b06e-9a4e3a09d43b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1ccf6e2-w9zhs\" (UID: \"d69cebfb-2af0-4064-b06e-9a4e3a09d43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf6e2-w9zhs" Oct 30 21:45:50.324216 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:50.324187 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d69cebfb-2af0-4064-b06e-9a4e3a09d43b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1ccf6e2-w9zhs\" (UID: \"d69cebfb-2af0-4064-b06e-9a4e3a09d43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf6e2-w9zhs" Oct 30 21:45:50.325828 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:50.325807 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d69cebfb-2af0-4064-b06e-9a4e3a09d43b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1ccf6e2-w9zhs\" (UID: \"d69cebfb-2af0-4064-b06e-9a4e3a09d43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf6e2-w9zhs" Oct 30 21:45:50.382662 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:50.382627 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hz8c\" (UniqueName: \"kubernetes.io/projected/d69cebfb-2af0-4064-b06e-9a4e3a09d43b-kube-api-access-2hz8c\") pod \"maas-api-revoke-sub-a1ccf6e2-w9zhs\" (UID: \"d69cebfb-2af0-4064-b06e-9a4e3a09d43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf6e2-w9zhs" Oct 30 21:45:50.483130 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:50.483099 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf6e2-w9zhs" Oct 30 21:45:50.853291 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:50.853182 4560 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:45:50.853291 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:50.853271 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" podUID="5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 21:45:50.856544 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:50.856518 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87abedcf-cshsf"] Oct 30 21:45:50.861596 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:50.861580 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87abedcf-cshsf" Oct 30 21:45:50.925837 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:50.925811 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf6e2-w9zhs"] Oct 30 21:45:50.957096 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:50.957064 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e03c1ba-6336-4b4e-aa29-f9700babe1fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87abedcf-cshsf\" (UID: \"5e03c1ba-6336-4b4e-aa29-f9700babe1fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87abedcf-cshsf" Oct 30 21:45:50.957258 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:50.957164 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlzzj\" (UniqueName: \"kubernetes.io/projected/5e03c1ba-6336-4b4e-aa29-f9700babe1fa-kube-api-access-wlzzj\") pod \"maas-api-revoke-sub-87abedcf-cshsf\" (UID: \"5e03c1ba-6336-4b4e-aa29-f9700babe1fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87abedcf-cshsf" Oct 30 21:45:50.957379 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:50.957365 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e03c1ba-6336-4b4e-aa29-f9700babe1fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87abedcf-cshsf\" (UID: \"5e03c1ba-6336-4b4e-aa29-f9700babe1fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87abedcf-cshsf" Oct 30 21:45:50.960230 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:50.959369 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87abedcf-cshsf"] Oct 30 21:45:50.962374 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:45:50.962317 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd69cebfb_2af0_4064_b06e_9a4e3a09d43b.slice/crio-484c6279482510c58ff7dd4b4caf58f96547cb96fd3ef2058a2f41f2485842e6 WatchSource:0}: Error finding container 484c6279482510c58ff7dd4b4caf58f96547cb96fd3ef2058a2f41f2485842e6: Status 404 returned error can't find the container with id 484c6279482510c58ff7dd4b4caf58f96547cb96fd3ef2058a2f41f2485842e6 Oct 30 21:45:51.058422 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:51.058386 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e03c1ba-6336-4b4e-aa29-f9700babe1fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87abedcf-cshsf\" (UID: \"5e03c1ba-6336-4b4e-aa29-f9700babe1fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87abedcf-cshsf" Oct 30 21:45:51.058565 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:51.058529 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlzzj\" (UniqueName: \"kubernetes.io/projected/5e03c1ba-6336-4b4e-aa29-f9700babe1fa-kube-api-access-wlzzj\") pod \"maas-api-revoke-sub-87abedcf-cshsf\" (UID: \"5e03c1ba-6336-4b4e-aa29-f9700babe1fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87abedcf-cshsf" Oct 30 21:45:51.058698 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:51.058683 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e03c1ba-6336-4b4e-aa29-f9700babe1fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87abedcf-cshsf\" (UID: \"5e03c1ba-6336-4b4e-aa29-f9700babe1fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87abedcf-cshsf" Oct 30 21:45:51.059138 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:51.059118 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e03c1ba-6336-4b4e-aa29-f9700babe1fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87abedcf-cshsf\" (UID: \"5e03c1ba-6336-4b4e-aa29-f9700babe1fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87abedcf-cshsf" Oct 30 21:45:51.061020 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:51.060997 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e03c1ba-6336-4b4e-aa29-f9700babe1fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87abedcf-cshsf\" (UID: \"5e03c1ba-6336-4b4e-aa29-f9700babe1fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87abedcf-cshsf" Oct 30 21:45:51.117486 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:51.117423 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlzzj\" (UniqueName: \"kubernetes.io/projected/5e03c1ba-6336-4b4e-aa29-f9700babe1fa-kube-api-access-wlzzj\") pod \"maas-api-revoke-sub-87abedcf-cshsf\" (UID: \"5e03c1ba-6336-4b4e-aa29-f9700babe1fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87abedcf-cshsf" Oct 30 21:45:51.175237 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:51.175211 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87abedcf-cshsf" Oct 30 21:45:51.555364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:51.554533 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4022d31-wlsbm"] Oct 30 21:45:51.559782 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:51.559755 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4022d31-wlsbm" Oct 30 21:45:51.645843 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:51.628396 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4022d31-wlsbm"] Oct 30 21:45:51.677498 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:51.675317 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87abedcf-cshsf"] Oct 30 21:45:51.687387 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:51.687116 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c13f33-cdc2-4f36-85c4-956b1955a361-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4022d31-wlsbm\" (UID: \"43c13f33-cdc2-4f36-85c4-956b1955a361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4022d31-wlsbm" Oct 30 21:45:51.687387 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:51.687200 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c13f33-cdc2-4f36-85c4-956b1955a361-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4022d31-wlsbm\" (UID: \"43c13f33-cdc2-4f36-85c4-956b1955a361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4022d31-wlsbm" Oct 30 21:45:51.687387 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:51.687225 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnpwn\" (UniqueName: \"kubernetes.io/projected/43c13f33-cdc2-4f36-85c4-956b1955a361-kube-api-access-dnpwn\") pod \"maas-api-revoke-sub-a4022d31-wlsbm\" (UID: \"43c13f33-cdc2-4f36-85c4-956b1955a361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4022d31-wlsbm" Oct 30 21:45:51.693952 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:45:51.693922 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e03c1ba_6336_4b4e_aa29_f9700babe1fa.slice/crio-f6dd45e6f2686782cac7608492351675451bf337fd7fd5ca7887255f0b11c573 WatchSource:0}: Error finding container f6dd45e6f2686782cac7608492351675451bf337fd7fd5ca7887255f0b11c573: Status 404 returned error can't find the container with id f6dd45e6f2686782cac7608492351675451bf337fd7fd5ca7887255f0b11c573 Oct 30 21:45:51.768507 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:51.768476 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87abedcf-cshsf" event={"ID":"5e03c1ba-6336-4b4e-aa29-f9700babe1fa","Type":"ContainerStarted","Data":"f6dd45e6f2686782cac7608492351675451bf337fd7fd5ca7887255f0b11c573"} Oct 30 21:45:51.770183 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:51.770157 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf6e2-w9zhs" event={"ID":"d69cebfb-2af0-4064-b06e-9a4e3a09d43b","Type":"ContainerStarted","Data":"e485c5a5bf9cc0a7aa4c1fd2652af6f3e110ec2bd56fc4cd68feffce114935c1"} Oct 30 21:45:51.770256 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:51.770204 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf6e2-w9zhs" event={"ID":"d69cebfb-2af0-4064-b06e-9a4e3a09d43b","Type":"ContainerStarted","Data":"484c6279482510c58ff7dd4b4caf58f96547cb96fd3ef2058a2f41f2485842e6"} Oct 30 21:45:51.788331 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:51.788298 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c13f33-cdc2-4f36-85c4-956b1955a361-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4022d31-wlsbm\" (UID: \"43c13f33-cdc2-4f36-85c4-956b1955a361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4022d31-wlsbm" Oct 30 21:45:51.788491 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:51.788463 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnpwn\" (UniqueName: \"kubernetes.io/projected/43c13f33-cdc2-4f36-85c4-956b1955a361-kube-api-access-dnpwn\") pod \"maas-api-revoke-sub-a4022d31-wlsbm\" (UID: \"43c13f33-cdc2-4f36-85c4-956b1955a361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4022d31-wlsbm" Oct 30 21:45:51.788675 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:51.788658 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c13f33-cdc2-4f36-85c4-956b1955a361-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4022d31-wlsbm\" (UID: \"43c13f33-cdc2-4f36-85c4-956b1955a361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4022d31-wlsbm" Oct 30 21:45:51.788995 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:51.788971 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c13f33-cdc2-4f36-85c4-956b1955a361-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4022d31-wlsbm\" (UID: \"43c13f33-cdc2-4f36-85c4-956b1955a361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4022d31-wlsbm" Oct 30 21:45:51.851966 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:51.851744 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c13f33-cdc2-4f36-85c4-956b1955a361-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4022d31-wlsbm\" (UID: \"43c13f33-cdc2-4f36-85c4-956b1955a361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4022d31-wlsbm" Oct 30 21:45:51.852138 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:51.851991 4560 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:45:51.852138 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:51.852057 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" podUID="5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 21:45:51.853800 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:51.853776 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnpwn\" (UniqueName: \"kubernetes.io/projected/43c13f33-cdc2-4f36-85c4-956b1955a361-kube-api-access-dnpwn\") pod \"maas-api-revoke-sub-a4022d31-wlsbm\" (UID: \"43c13f33-cdc2-4f36-85c4-956b1955a361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4022d31-wlsbm" Oct 30 21:45:51.886208 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:51.886163 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf6e2-w9zhs" podStartSLOduration=1.8861504390000001 podStartE2EDuration="1.886150439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:51.884744866 +0000 UTC m=+1927.439444642" watchObservedRunningTime="2026-10-30 21:45:51.886150439 +0000 UTC m=+1927.440850253" Oct 30 21:45:51.978403 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:51.978377 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4022d31-wlsbm" Oct 30 21:45:52.059880 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:52.059698 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78166b53-wjpvh"] Oct 30 21:45:52.071834 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:52.071798 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78166b53-wjpvh"] Oct 30 21:45:52.388253 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:52.388222 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4022d31-wlsbm"] Oct 30 21:45:52.391599 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:45:52.391556 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c13f33_cdc2_4f36_85c4_956b1955a361.slice/crio-3cab2e4ecced3b56176705e753c18ed2478277efad947cd0b264fa35cb0c3d5b WatchSource:0}: Error finding container 3cab2e4ecced3b56176705e753c18ed2478277efad947cd0b264fa35cb0c3d5b: Status 404 returned error can't find the container with id 3cab2e4ecced3b56176705e753c18ed2478277efad947cd0b264fa35cb0c3d5b Oct 30 21:45:52.577849 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:52.577822 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50c05596-08fd-4153-9de2-473ac0db79f2" path="/var/lib/kubelet/pods/50c05596-08fd-4153-9de2-473ac0db79f2/volumes" Oct 30 21:45:52.777921 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:52.777886 4560 generic.go:356] "Generic (PLEG): container finished" podID="d69cebfb-2af0-4064-b06e-9a4e3a09d43b" containerID="e485c5a5bf9cc0a7aa4c1fd2652af6f3e110ec2bd56fc4cd68feffce114935c1" exitCode=0 Oct 30 21:45:52.778060 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:52.777940 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf6e2-w9zhs" event={"ID":"d69cebfb-2af0-4064-b06e-9a4e3a09d43b","Type":"ContainerDied","Data":"e485c5a5bf9cc0a7aa4c1fd2652af6f3e110ec2bd56fc4cd68feffce114935c1"} Oct 30 21:45:52.779598 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:52.779575 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87abedcf-cshsf" event={"ID":"5e03c1ba-6336-4b4e-aa29-f9700babe1fa","Type":"ContainerStarted","Data":"7815a81d8ba36e1456d4cfc054ca7353c44a8041754bceb5dc58ca97ebb0ceaf"} Oct 30 21:45:52.781025 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:52.781007 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4022d31-wlsbm" event={"ID":"43c13f33-cdc2-4f36-85c4-956b1955a361","Type":"ContainerStarted","Data":"91cd6fb03776777d6566ef01ab27150deb955f2b05992d6714b630f8a0cc44b3"} Oct 30 21:45:52.781106 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:52.781039 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4022d31-wlsbm" event={"ID":"43c13f33-cdc2-4f36-85c4-956b1955a361","Type":"ContainerStarted","Data":"3cab2e4ecced3b56176705e753c18ed2478277efad947cd0b264fa35cb0c3d5b"} Oct 30 21:45:52.848790 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:52.848757 4560 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:45:52.849006 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:52.848980 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" podUID="5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 21:45:53.054401 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:53.053459 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4022d31-wlsbm" podStartSLOduration=2.053439262 podStartE2EDuration="2.053439262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:53.044941895 +0000 UTC m=+1928.599641673" watchObservedRunningTime="2026-10-30 21:45:53.053439262 +0000 UTC m=+1928.608139039" Oct 30 21:45:53.158851 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:53.158797 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87abedcf-cshsf" podStartSLOduration=3.158783062 podStartE2EDuration="3.158783062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:53.155870935 +0000 UTC m=+1928.710570710" watchObservedRunningTime="2026-10-30 21:45:53.158783062 +0000 UTC m=+1928.713482837" Oct 30 21:45:53.788725 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:53.788693 4560 generic.go:356] "Generic (PLEG): container finished" podID="5e03c1ba-6336-4b4e-aa29-f9700babe1fa" containerID="7815a81d8ba36e1456d4cfc054ca7353c44a8041754bceb5dc58ca97ebb0ceaf" exitCode=0 Oct 30 21:45:53.788901 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:53.788745 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87abedcf-cshsf" event={"ID":"5e03c1ba-6336-4b4e-aa29-f9700babe1fa","Type":"ContainerDied","Data":"7815a81d8ba36e1456d4cfc054ca7353c44a8041754bceb5dc58ca97ebb0ceaf"} Oct 30 21:45:53.790640 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:53.790622 4560 generic.go:356] "Generic (PLEG): container finished" podID="43c13f33-cdc2-4f36-85c4-956b1955a361" containerID="91cd6fb03776777d6566ef01ab27150deb955f2b05992d6714b630f8a0cc44b3" exitCode=0 Oct 30 21:45:53.790713 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:53.790693 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4022d31-wlsbm" event={"ID":"43c13f33-cdc2-4f36-85c4-956b1955a361","Type":"ContainerDied","Data":"91cd6fb03776777d6566ef01ab27150deb955f2b05992d6714b630f8a0cc44b3"} Oct 30 21:45:53.854363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:53.854009 4560 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:45:53.854363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:53.854083 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" podUID="5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 21:45:53.984426 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:53.984404 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf6e2-w9zhs" Oct 30 21:45:54.026283 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:54.026258 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d69cebfb-2af0-4064-b06e-9a4e3a09d43b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d69cebfb-2af0-4064-b06e-9a4e3a09d43b-maas-api-xks-ca\") pod \"d69cebfb-2af0-4064-b06e-9a4e3a09d43b\" (UID: \"d69cebfb-2af0-4064-b06e-9a4e3a09d43b\") " Oct 30 21:45:54.026445 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:54.026298 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d69cebfb-2af0-4064-b06e-9a4e3a09d43b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d69cebfb-2af0-4064-b06e-9a4e3a09d43b-maas-api-openshift-ca\") pod \"d69cebfb-2af0-4064-b06e-9a4e3a09d43b\" (UID: \"d69cebfb-2af0-4064-b06e-9a4e3a09d43b\") " Oct 30 21:45:54.048421 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:54.046736 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d69cebfb-2af0-4064-b06e-9a4e3a09d43b-maas-api-openshift-ca" pod "d69cebfb-2af0-4064-b06e-9a4e3a09d43b" (UID: "d69cebfb-2af0-4064-b06e-9a4e3a09d43b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:54.048555 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:54.048426 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d69cebfb-2af0-4064-b06e-9a4e3a09d43b-kube-api-access-2hz8c\" (UniqueName: \"kubernetes.io/projected/d69cebfb-2af0-4064-b06e-9a4e3a09d43b-kube-api-access-2hz8c\") pod \"d69cebfb-2af0-4064-b06e-9a4e3a09d43b\" (UID: \"d69cebfb-2af0-4064-b06e-9a4e3a09d43b\") " Oct 30 21:45:54.052381 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:54.049058 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d69cebfb-2af0-4064-b06e-9a4e3a09d43b-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:54.053277 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:54.053193 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d69cebfb-2af0-4064-b06e-9a4e3a09d43b-kube-api-access-2hz8c" pod "d69cebfb-2af0-4064-b06e-9a4e3a09d43b" (UID: "d69cebfb-2af0-4064-b06e-9a4e3a09d43b"). InnerVolumeSpecName "kube-api-access-2hz8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:54.054426 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:54.054406 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d69cebfb-2af0-4064-b06e-9a4e3a09d43b-maas-api-xks-ca" pod "d69cebfb-2af0-4064-b06e-9a4e3a09d43b" (UID: "d69cebfb-2af0-4064-b06e-9a4e3a09d43b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:54.151365 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:54.150369 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d69cebfb-2af0-4064-b06e-9a4e3a09d43b-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:54.151365 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:54.150423 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hz8c\" (UniqueName: \"kubernetes.io/projected/d69cebfb-2af0-4064-b06e-9a4e3a09d43b-kube-api-access-2hz8c\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:54.800029 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:54.799991 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf6e2-w9zhs" Oct 30 21:45:54.800293 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:54.800266 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf6e2-w9zhs" event={"ID":"d69cebfb-2af0-4064-b06e-9a4e3a09d43b","Type":"ContainerDied","Data":"484c6279482510c58ff7dd4b4caf58f96547cb96fd3ef2058a2f41f2485842e6"} Oct 30 21:45:54.800371 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:54.800320 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="484c6279482510c58ff7dd4b4caf58f96547cb96fd3ef2058a2f41f2485842e6" Oct 30 21:45:54.851825 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:54.851741 4560 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:45:54.852123 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:54.852082 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" podUID="5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 21:45:55.005526 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:55.005492 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87abedcf-cshsf" Oct 30 21:45:55.062150 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:55.062124 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8647ba4f-x56gq"] Oct 30 21:45:55.062949 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:55.062926 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4022d31-wlsbm" Oct 30 21:45:55.065154 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:55.065135 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e03c1ba-6336-4b4e-aa29-f9700babe1fa-kube-api-access-wlzzj\" (UniqueName: \"kubernetes.io/projected/5e03c1ba-6336-4b4e-aa29-f9700babe1fa-kube-api-access-wlzzj\") pod \"5e03c1ba-6336-4b4e-aa29-f9700babe1fa\" (UID: \"5e03c1ba-6336-4b4e-aa29-f9700babe1fa\") " Oct 30 21:45:55.065279 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:55.065223 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e03c1ba-6336-4b4e-aa29-f9700babe1fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e03c1ba-6336-4b4e-aa29-f9700babe1fa-maas-api-openshift-ca\") pod \"5e03c1ba-6336-4b4e-aa29-f9700babe1fa\" (UID: \"5e03c1ba-6336-4b4e-aa29-f9700babe1fa\") " Oct 30 21:45:55.065386 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:55.065372 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e03c1ba-6336-4b4e-aa29-f9700babe1fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e03c1ba-6336-4b4e-aa29-f9700babe1fa-maas-api-xks-ca\") pod \"5e03c1ba-6336-4b4e-aa29-f9700babe1fa\" (UID: \"5e03c1ba-6336-4b4e-aa29-f9700babe1fa\") " Oct 30 21:45:55.065745 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:55.065727 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e03c1ba-6336-4b4e-aa29-f9700babe1fa-maas-api-openshift-ca" pod "5e03c1ba-6336-4b4e-aa29-f9700babe1fa" (UID: "5e03c1ba-6336-4b4e-aa29-f9700babe1fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:55.067280 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:55.067261 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e03c1ba-6336-4b4e-aa29-f9700babe1fa-kube-api-access-wlzzj" pod "5e03c1ba-6336-4b4e-aa29-f9700babe1fa" (UID: "5e03c1ba-6336-4b4e-aa29-f9700babe1fa"). InnerVolumeSpecName "kube-api-access-wlzzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:55.067364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:55.067296 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e03c1ba-6336-4b4e-aa29-f9700babe1fa-maas-api-xks-ca" pod "5e03c1ba-6336-4b4e-aa29-f9700babe1fa" (UID: "5e03c1ba-6336-4b4e-aa29-f9700babe1fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:55.107008 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:55.106960 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8647ba4f-x56gq"] Oct 30 21:45:55.167133 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:55.167084 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43c13f33-cdc2-4f36-85c4-956b1955a361-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c13f33-cdc2-4f36-85c4-956b1955a361-maas-api-xks-ca\") pod \"43c13f33-cdc2-4f36-85c4-956b1955a361\" (UID: \"43c13f33-cdc2-4f36-85c4-956b1955a361\") " Oct 30 21:45:55.167329 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:55.167195 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43c13f33-cdc2-4f36-85c4-956b1955a361-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c13f33-cdc2-4f36-85c4-956b1955a361-maas-api-openshift-ca\") pod \"43c13f33-cdc2-4f36-85c4-956b1955a361\" (UID: \"43c13f33-cdc2-4f36-85c4-956b1955a361\") " Oct 30 21:45:55.167329 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:55.167298 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43c13f33-cdc2-4f36-85c4-956b1955a361-kube-api-access-dnpwn\" (UniqueName: \"kubernetes.io/projected/43c13f33-cdc2-4f36-85c4-956b1955a361-kube-api-access-dnpwn\") pod \"43c13f33-cdc2-4f36-85c4-956b1955a361\" (UID: \"43c13f33-cdc2-4f36-85c4-956b1955a361\") " Oct 30 21:45:55.167648 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:55.167619 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43c13f33-cdc2-4f36-85c4-956b1955a361-maas-api-openshift-ca" pod "43c13f33-cdc2-4f36-85c4-956b1955a361" (UID: "43c13f33-cdc2-4f36-85c4-956b1955a361"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:55.168090 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:55.168062 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e03c1ba-6336-4b4e-aa29-f9700babe1fa-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:55.168184 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:55.168094 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e03c1ba-6336-4b4e-aa29-f9700babe1fa-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:55.168184 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:55.168109 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c13f33-cdc2-4f36-85c4-956b1955a361-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:55.168184 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:55.168123 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlzzj\" (UniqueName: \"kubernetes.io/projected/5e03c1ba-6336-4b4e-aa29-f9700babe1fa-kube-api-access-wlzzj\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:55.169996 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:55.169968 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43c13f33-cdc2-4f36-85c4-956b1955a361-maas-api-xks-ca" pod "43c13f33-cdc2-4f36-85c4-956b1955a361" (UID: "43c13f33-cdc2-4f36-85c4-956b1955a361"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:55.170493 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:55.170464 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c13f33-cdc2-4f36-85c4-956b1955a361-kube-api-access-dnpwn" pod "43c13f33-cdc2-4f36-85c4-956b1955a361" (UID: "43c13f33-cdc2-4f36-85c4-956b1955a361"). InnerVolumeSpecName "kube-api-access-dnpwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:55.269609 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:55.269572 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnpwn\" (UniqueName: \"kubernetes.io/projected/43c13f33-cdc2-4f36-85c4-956b1955a361-kube-api-access-dnpwn\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:55.269609 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:55.269605 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c13f33-cdc2-4f36-85c4-956b1955a361-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:55.809193 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:55.809160 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87abedcf-cshsf" Oct 30 21:45:55.809193 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:55.809176 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87abedcf-cshsf" event={"ID":"5e03c1ba-6336-4b4e-aa29-f9700babe1fa","Type":"ContainerDied","Data":"f6dd45e6f2686782cac7608492351675451bf337fd7fd5ca7887255f0b11c573"} Oct 30 21:45:55.809460 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:55.809225 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6dd45e6f2686782cac7608492351675451bf337fd7fd5ca7887255f0b11c573" Oct 30 21:45:55.810540 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:55.810516 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4022d31-wlsbm" event={"ID":"43c13f33-cdc2-4f36-85c4-956b1955a361","Type":"ContainerDied","Data":"3cab2e4ecced3b56176705e753c18ed2478277efad947cd0b264fa35cb0c3d5b"} Oct 30 21:45:55.810654 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:55.810558 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cab2e4ecced3b56176705e753c18ed2478277efad947cd0b264fa35cb0c3d5b" Oct 30 21:45:55.810654 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:55.810571 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4022d31-wlsbm" Oct 30 21:45:55.848435 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:55.848410 4560 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:45:55.848520 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:55.848454 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" podUID="5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 21:45:56.599278 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:56.599239 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93390da6-be64-4eb2-a121-f90e7bbc4775" path="/var/lib/kubelet/pods/93390da6-be64-4eb2-a121-f90e7bbc4775/volumes" Oct 30 21:45:56.852882 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:56.852807 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" Oct 30 21:45:56.853220 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:56.853198 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" Oct 30 21:45:56.853736 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:45:56.853717 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" Oct 30 21:46:00.905612 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:00.905579 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-157f1e1f-2nx6l"] Oct 30 21:46:00.906163 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:00.906146 4560 cpu_manager.go:414] "containerMap: removing container" podUID="43c13f33-cdc2-4f36-85c4-956b1955a361" containerName="revoke-keys" Oct 30 21:46:00.906207 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:00.906165 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43c13f33-cdc2-4f36-85c4-956b1955a361" containerName="revoke-keys" Oct 30 21:46:00.906207 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:00.906178 4560 cpu_manager.go:414] "containerMap: removing container" podUID="5e03c1ba-6336-4b4e-aa29-f9700babe1fa" containerName="revoke-keys" Oct 30 21:46:00.906207 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:00.906182 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e03c1ba-6336-4b4e-aa29-f9700babe1fa" containerName="revoke-keys" Oct 30 21:46:00.906207 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:00.906205 4560 cpu_manager.go:414] "containerMap: removing container" podUID="d69cebfb-2af0-4064-b06e-9a4e3a09d43b" containerName="revoke-keys" Oct 30 21:46:00.906324 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:00.906210 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d69cebfb-2af0-4064-b06e-9a4e3a09d43b" containerName="revoke-keys" Oct 30 21:46:00.906324 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:00.906289 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="43c13f33-cdc2-4f36-85c4-956b1955a361" containerName="revoke-keys" Oct 30 21:46:00.906324 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:00.906297 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e03c1ba-6336-4b4e-aa29-f9700babe1fa" containerName="revoke-keys" Oct 30 21:46:00.906324 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:00.906306 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="d69cebfb-2af0-4064-b06e-9a4e3a09d43b" containerName="revoke-keys" Oct 30 21:46:00.910919 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:00.910901 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157f1e1f-2nx6l" Oct 30 21:46:00.967425 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:00.967391 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-157f1e1f-2nx6l"] Oct 30 21:46:01.008295 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:01.008266 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:46:01.008295 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:01.008272 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:46:01.008512 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:01.008299 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:46:01.008512 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:01.008334 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:46:01.053201 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:01.053172 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zqtz\" (UniqueName: \"kubernetes.io/projected/0dbb0932-e12b-4768-861b-4832d25dd952-kube-api-access-2zqtz\") pod \"maas-api-revoke-sub-157f1e1f-2nx6l\" (UID: \"0dbb0932-e12b-4768-861b-4832d25dd952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157f1e1f-2nx6l" Oct 30 21:46:01.053366 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:01.053258 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dbb0932-e12b-4768-861b-4832d25dd952-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-157f1e1f-2nx6l\" (UID: \"0dbb0932-e12b-4768-861b-4832d25dd952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157f1e1f-2nx6l" Oct 30 21:46:01.053544 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:01.053525 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dbb0932-e12b-4768-861b-4832d25dd952-maas-api-xks-ca\") pod \"maas-api-revoke-sub-157f1e1f-2nx6l\" (UID: \"0dbb0932-e12b-4768-861b-4832d25dd952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157f1e1f-2nx6l" Oct 30 21:46:01.154243 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:01.154216 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dbb0932-e12b-4768-861b-4832d25dd952-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-157f1e1f-2nx6l\" (UID: \"0dbb0932-e12b-4768-861b-4832d25dd952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157f1e1f-2nx6l" Oct 30 21:46:01.154449 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:01.154386 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dbb0932-e12b-4768-861b-4832d25dd952-maas-api-xks-ca\") pod \"maas-api-revoke-sub-157f1e1f-2nx6l\" (UID: \"0dbb0932-e12b-4768-861b-4832d25dd952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157f1e1f-2nx6l" Oct 30 21:46:01.154526 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:01.154476 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zqtz\" (UniqueName: \"kubernetes.io/projected/0dbb0932-e12b-4768-861b-4832d25dd952-kube-api-access-2zqtz\") pod \"maas-api-revoke-sub-157f1e1f-2nx6l\" (UID: \"0dbb0932-e12b-4768-861b-4832d25dd952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157f1e1f-2nx6l" Oct 30 21:46:01.154945 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:01.154921 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dbb0932-e12b-4768-861b-4832d25dd952-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-157f1e1f-2nx6l\" (UID: \"0dbb0932-e12b-4768-861b-4832d25dd952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157f1e1f-2nx6l" Oct 30 21:46:01.156671 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:01.156628 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dbb0932-e12b-4768-861b-4832d25dd952-maas-api-xks-ca\") pod \"maas-api-revoke-sub-157f1e1f-2nx6l\" (UID: \"0dbb0932-e12b-4768-861b-4832d25dd952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157f1e1f-2nx6l" Oct 30 21:46:01.212731 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:01.212703 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zqtz\" (UniqueName: \"kubernetes.io/projected/0dbb0932-e12b-4768-861b-4832d25dd952-kube-api-access-2zqtz\") pod \"maas-api-revoke-sub-157f1e1f-2nx6l\" (UID: \"0dbb0932-e12b-4768-861b-4832d25dd952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157f1e1f-2nx6l" Oct 30 21:46:01.223544 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:01.223522 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157f1e1f-2nx6l" Oct 30 21:46:01.408154 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:01.408068 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:46:01.608241 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:01.608209 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-157f1e1f-2nx6l"] Oct 30 21:46:01.613121 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:46:01.613098 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dbb0932_e12b_4768_861b_4832d25dd952.slice/crio-bc93431044390dcfac81a086e7f15a2acb16df53f24d910d5f8f7b16374cd493 WatchSource:0}: Error finding container bc93431044390dcfac81a086e7f15a2acb16df53f24d910d5f8f7b16374cd493: Status 404 returned error can't find the container with id bc93431044390dcfac81a086e7f15a2acb16df53f24d910d5f8f7b16374cd493 Oct 30 21:46:01.855692 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:01.855628 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157f1e1f-2nx6l" event={"ID":"0dbb0932-e12b-4768-861b-4832d25dd952","Type":"ContainerStarted","Data":"bc93431044390dcfac81a086e7f15a2acb16df53f24d910d5f8f7b16374cd493"} Oct 30 21:46:02.532363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:02.531398 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw"] Oct 30 21:46:02.532363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:02.531637 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" podUID="34448c97-8c89-4c69-9e3b-8f513d88b803" containerName="istio-proxy" containerID="cri-o://8944a3b4f6fbac741bd92c01c7ac681a52de24ddde7774811b24688f15708db0" gracePeriod=30 Oct 30 21:46:03.312826 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:03.312724 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157f1e1f-2nx6l" event={"ID":"0dbb0932-e12b-4768-861b-4832d25dd952","Type":"ContainerStarted","Data":"35ab5f78cd38aaa19f25027fa74edafef9281ecc96f8447bd39f4d49ed9601d5"} Oct 30 21:46:03.331920 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:03.331887 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c9bafde-5lfrj"] Oct 30 21:46:03.831422 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:03.815451 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" podUID="9e2d7535-075d-4211-b814-35d6abff7605" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:03.831422 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:03.815487 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" podUID="4dee11ef-9647-47ee-a4b1-686680256542" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:03.831422 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:03.816594 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" podUID="0155a02e-8df7-4847-b40f-b8d9e4bac8c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded" Oct 30 21:46:03.923446 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:03.923400 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c9bafde-5lfrj"] Oct 30 21:46:03.929668 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:03.929603 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157f1e1f-2nx6l" podStartSLOduration=3.92958161 podStartE2EDuration="3.92958161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:03.820788448 +0000 UTC m=+1939.375488226" watchObservedRunningTime="2026-10-30 21:46:03.92958161 +0000 UTC m=+1939.484281386" Oct 30 21:46:04.018362 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:04.017427 4560 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" podUID="4dee11ef-9647-47ee-a4b1-686680256542" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:04.074585 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:04.070153 4560 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" podUID="b80b0fba-3830-4a74-a724-aead35443b89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": context deadline exceeded" Oct 30 21:46:04.110409 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:04.109571 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" podUID="45a5bac0-28ab-4c78-856e-303f6d4522de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:04.244519 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:04.244478 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" podUID="971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:04.245064 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:04.245037 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" podUID="e974bae1-bb18-46b0-a4ee-4c01d6455e32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 21:46:04.272637 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:04.272589 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" podUID="8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:04.355364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:04.350593 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" podUID="90bfb1e4-9526-40f6-aed4-ee46c2d86471" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:04.368468 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:04.367824 4560 generic.go:356] "Generic (PLEG): container finished" podID="0dbb0932-e12b-4768-861b-4832d25dd952" containerID="35ab5f78cd38aaa19f25027fa74edafef9281ecc96f8447bd39f4d49ed9601d5" exitCode=0 Oct 30 21:46:04.368468 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:04.367955 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157f1e1f-2nx6l" event={"ID":"0dbb0932-e12b-4768-861b-4832d25dd952","Type":"ContainerDied","Data":"35ab5f78cd38aaa19f25027fa74edafef9281ecc96f8447bd39f4d49ed9601d5"} Oct 30 21:46:04.378896 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:04.378853 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" podUID="0ff8ae26-4364-4c4e-9285-721d1a486a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:04.751473 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:04.722913 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" podUID="e3281dbe-e8f9-4bab-9ae4-2279e79b7292" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 30 21:46:04.751473 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:04.723942 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" podUID="a59a0fc3-8316-47eb-a42e-552d202b1ab7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:04.793826 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:04.793779 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c4b67bd-02b1-4938-a169-9a08d0ac3530" path="/var/lib/kubelet/pods/5c4b67bd-02b1-4938-a169-9a08d0ac3530/volumes" Oct 30 21:46:04.828501 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:04.828449 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" podUID="0155a02e-8df7-4847-b40f-b8d9e4bac8c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded" Oct 30 21:46:04.829057 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:04.829027 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" podUID="4dee11ef-9647-47ee-a4b1-686680256542" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Oct 30 21:46:04.829572 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:04.829552 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" Oct 30 21:46:04.829760 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:04.829744 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" Oct 30 21:46:04.831839 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:04.831809 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" podUID="9e2d7535-075d-4211-b814-35d6abff7605" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:04.832609 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:04.832592 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" Oct 30 21:46:05.158422 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:05.141489 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" podUID="45a5bac0-28ab-4c78-856e-303f6d4522de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:05.172489 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:05.158735 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" Oct 30 21:46:05.310366 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:05.305438 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" podUID="8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:05.310366 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:05.306472 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" Oct 30 21:46:05.457290 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:05.457186 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" podUID="e974bae1-bb18-46b0-a4ee-4c01d6455e32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 21:46:05.458216 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:05.458190 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" Oct 30 21:46:05.487481 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:05.483580 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" podUID="90bfb1e4-9526-40f6-aed4-ee46c2d86471" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Oct 30 21:46:05.487481 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:05.483935 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" Oct 30 21:46:05.734417 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:05.731425 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" podUID="e3281dbe-e8f9-4bab-9ae4-2279e79b7292" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:05.734417 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:05.732122 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" Oct 30 21:46:05.734417 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:05.733010 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" podUID="a59a0fc3-8316-47eb-a42e-552d202b1ab7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 30 21:46:05.734417 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:05.733356 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" Oct 30 21:46:05.753582 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:05.753540 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" podUID="b80b0fba-3830-4a74-a724-aead35443b89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:05.839514 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:05.839472 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" podUID="9e2d7535-075d-4211-b814-35d6abff7605" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:46:05.869368 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:05.840882 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" podUID="4dee11ef-9647-47ee-a4b1-686680256542" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:05.896383 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:05.845451 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" podUID="0155a02e-8df7-4847-b40f-b8d9e4bac8c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:06.132513 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:06.132423 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" podUID="27df5dfe-47a4-4ebb-bd09-1ceef406600a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:06.158364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:06.152500 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" podUID="45a5bac0-28ab-4c78-856e-303f6d4522de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:06.222580 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:06.222486 4560 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" podUID="8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:06.235494 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:06.235451 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" podUID="971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:06.321601 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:06.321527 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" podUID="8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:06.437903 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:06.437803 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" podUID="1e044c02-0053-40e8-9f80-e806cd2d9ae5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:06.465987 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:06.465950 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" podUID="e974bae1-bb18-46b0-a4ee-4c01d6455e32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 21:46:06.467810 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:06.467767 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" podUID="0ff8ae26-4364-4c4e-9285-721d1a486a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:06.498636 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:06.498592 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" podUID="90bfb1e4-9526-40f6-aed4-ee46c2d86471" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:06.790367 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:06.778859 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" podUID="b80b0fba-3830-4a74-a724-aead35443b89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 30 21:46:06.790367 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:06.781923 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" podUID="a59a0fc3-8316-47eb-a42e-552d202b1ab7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:06.790367 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:06.782848 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" podUID="e3281dbe-e8f9-4bab-9ae4-2279e79b7292" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 30 21:46:06.891361 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:06.864204 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" podUID="9e2d7535-075d-4211-b814-35d6abff7605" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:06.922368 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:06.922237 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" Oct 30 21:46:06.925964 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:06.925931 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" podUID="0155a02e-8df7-4847-b40f-b8d9e4bac8c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:07.088745 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:07.088657 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" Oct 30 21:46:07.141430 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:07.141391 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" podUID="27df5dfe-47a4-4ebb-bd09-1ceef406600a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:07.186367 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:07.143299 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" Oct 30 21:46:07.187815 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:07.159442 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" podUID="45a5bac0-28ab-4c78-856e-303f6d4522de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": context deadline exceeded" Oct 30 21:46:07.211360 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:07.210432 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" podUID="4dee11ef-9647-47ee-a4b1-686680256542" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:07.259510 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:07.259468 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" podUID="971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:07.261428 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:07.261405 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" Oct 30 21:46:07.330461 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:07.330419 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" podUID="8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:07.475520 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:07.475477 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" podUID="e974bae1-bb18-46b0-a4ee-4c01d6455e32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 21:46:07.493050 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:07.479006 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" podUID="1e044c02-0053-40e8-9f80-e806cd2d9ae5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:07.493050 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:07.479046 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" podUID="0ff8ae26-4364-4c4e-9285-721d1a486a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:07.493050 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:07.480796 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" Oct 30 21:46:07.493050 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:07.480852 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" Oct 30 21:46:07.517143 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:07.517095 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" podUID="90bfb1e4-9526-40f6-aed4-ee46c2d86471" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:07.828491 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:07.828440 4560 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" podUID="0ff8ae26-4364-4c4e-9285-721d1a486a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": context deadline exceeded" Oct 30 21:46:07.940281 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:07.940236 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" podUID="b80b0fba-3830-4a74-a724-aead35443b89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:08.342416 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:08.339283 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" podUID="27df5dfe-47a4-4ebb-bd09-1ceef406600a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:08.342416 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:07.989896 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" podUID="e3281dbe-e8f9-4bab-9ae4-2279e79b7292" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:08.342416 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:07.992122 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" podUID="0155a02e-8df7-4847-b40f-b8d9e4bac8c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:08.484363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:07.989760 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" podUID="a59a0fc3-8316-47eb-a42e-552d202b1ab7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:08.535411 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:08.517709 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" podUID="90bfb1e4-9526-40f6-aed4-ee46c2d86471" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:08.548988 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:08.548945 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" podUID="45a5bac0-28ab-4c78-856e-303f6d4522de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:08.549747 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:08.549716 4560 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" podUID="27df5dfe-47a4-4ebb-bd09-1ceef406600a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:08.550317 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:08.550286 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" podUID="0ff8ae26-4364-4c4e-9285-721d1a486a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:08.554726 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:08.554700 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" podUID="e974bae1-bb18-46b0-a4ee-4c01d6455e32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:08.555114 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:08.555090 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" podUID="1e044c02-0053-40e8-9f80-e806cd2d9ae5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 30 21:46:08.565711 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:08.565672 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" podUID="4dee11ef-9647-47ee-a4b1-686680256542" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:08.567038 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:08.567009 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" podUID="971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:09.006767 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:09.006506 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" podUID="9e2d7535-075d-4211-b814-35d6abff7605" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:09.302361 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:09.298924 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" podUID="8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 30 21:46:09.495081 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:09.495039 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pcphw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:46:09.495377 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:09.495321 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" podUID="34448c97-8c89-4c69-9e3b-8f513d88b803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 21:46:09.543806 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:09.543766 4560 generic.go:356] "Generic (PLEG): container finished" podID="34448c97-8c89-4c69-9e3b-8f513d88b803" containerID="8944a3b4f6fbac741bd92c01c7ac681a52de24ddde7774811b24688f15708db0" exitCode=0 Oct 30 21:46:09.572332 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:09.572236 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" podUID="b80b0fba-3830-4a74-a724-aead35443b89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 30 21:46:09.655659 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:09.655629 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157f1e1f-2nx6l" Oct 30 21:46:09.693052 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:09.659309 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" podUID="27df5dfe-47a4-4ebb-bd09-1ceef406600a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded" Oct 30 21:46:09.695864 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:09.659373 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" podUID="e3281dbe-e8f9-4bab-9ae4-2279e79b7292" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:09.728260 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:09.659396 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" podUID="0155a02e-8df7-4847-b40f-b8d9e4bac8c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded" Oct 30 21:46:09.728836 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:09.666249 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" podUID="971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:09.729625 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:09.675749 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" podUID="0ff8ae26-4364-4c4e-9285-721d1a486a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 21:46:09.741398 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:09.680517 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" podUID="a59a0fc3-8316-47eb-a42e-552d202b1ab7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 30 21:46:09.742999 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:09.680570 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" podUID="90bfb1e4-9526-40f6-aed4-ee46c2d86471" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Oct 30 21:46:09.743378 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:09.680591 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" podUID="45a5bac0-28ab-4c78-856e-303f6d4522de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": context deadline exceeded" Oct 30 21:46:09.744165 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:09.686541 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" podUID="1e044c02-0053-40e8-9f80-e806cd2d9ae5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:09.744876 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:09.686573 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" podUID="e974bae1-bb18-46b0-a4ee-4c01d6455e32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 21:46:09.746507 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:09.686600 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" podUID="4dee11ef-9647-47ee-a4b1-686680256542" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:10.054380 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:10.019651 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dbb0932-e12b-4768-861b-4832d25dd952-kube-api-access-2zqtz\" (UniqueName: \"kubernetes.io/projected/0dbb0932-e12b-4768-861b-4832d25dd952-kube-api-access-2zqtz\") pod \"0dbb0932-e12b-4768-861b-4832d25dd952\" (UID: \"0dbb0932-e12b-4768-861b-4832d25dd952\") " Oct 30 21:46:10.055103 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:10.055067 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dbb0932-e12b-4768-861b-4832d25dd952-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dbb0932-e12b-4768-861b-4832d25dd952-maas-api-xks-ca\") pod \"0dbb0932-e12b-4768-861b-4832d25dd952\" (UID: \"0dbb0932-e12b-4768-861b-4832d25dd952\") " Oct 30 21:46:10.055408 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:10.055378 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dbb0932-e12b-4768-861b-4832d25dd952-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dbb0932-e12b-4768-861b-4832d25dd952-maas-api-openshift-ca\") pod \"0dbb0932-e12b-4768-861b-4832d25dd952\" (UID: \"0dbb0932-e12b-4768-861b-4832d25dd952\") " Oct 30 21:46:10.092413 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:10.078024 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dbb0932-e12b-4768-861b-4832d25dd952-maas-api-openshift-ca" pod "0dbb0932-e12b-4768-861b-4832d25dd952" (UID: "0dbb0932-e12b-4768-861b-4832d25dd952"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:10.139194 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:10.128584 4560 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" podUID="9e2d7535-075d-4211-b814-35d6abff7605" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": context deadline exceeded" Oct 30 21:46:10.147413 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:10.135509 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dbb0932-e12b-4768-861b-4832d25dd952-kube-api-access-2zqtz" pod "0dbb0932-e12b-4768-861b-4832d25dd952" (UID: "0dbb0932-e12b-4768-861b-4832d25dd952"). InnerVolumeSpecName "kube-api-access-2zqtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:10.179612 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:10.163513 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dbb0932-e12b-4768-861b-4832d25dd952-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.179612 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:10.163551 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zqtz\" (UniqueName: \"kubernetes.io/projected/0dbb0932-e12b-4768-861b-4832d25dd952-kube-api-access-2zqtz\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.241364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:10.240646 4560 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" podUID="90bfb1e4-9526-40f6-aed4-ee46c2d86471" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:10.382794 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:10.381710 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" podUID="8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 30 21:46:10.487413 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:10.466186 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" podUID="9e2d7535-075d-4211-b814-35d6abff7605" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:46:10.499685 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:10.492160 4560 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" podUID="a59a0fc3-8316-47eb-a42e-552d202b1ab7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": context deadline exceeded" Oct 30 21:46:10.499685 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:10.493304 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dbb0932-e12b-4768-861b-4832d25dd952-maas-api-xks-ca" pod "0dbb0932-e12b-4768-861b-4832d25dd952" (UID: "0dbb0932-e12b-4768-861b-4832d25dd952"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:10.584835 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:46:10.584803 4560 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.01s" Oct 30 21:46:10.585056 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:10.585034 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" event={"ID":"34448c97-8c89-4c69-9e3b-8f513d88b803","Type":"ContainerDied","Data":"8944a3b4f6fbac741bd92c01c7ac681a52de24ddde7774811b24688f15708db0"} Oct 30 21:46:10.643359 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:10.636563 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dbb0932-e12b-4768-861b-4832d25dd952-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.718508 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:10.718468 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" podUID="b80b0fba-3830-4a74-a724-aead35443b89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:10.770658 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:10.766430 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" podUID="4dee11ef-9647-47ee-a4b1-686680256542" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:10.930363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:10.921324 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157f1e1f-2nx6l" event={"ID":"0dbb0932-e12b-4768-861b-4832d25dd952","Type":"ContainerDied","Data":"bc93431044390dcfac81a086e7f15a2acb16df53f24d910d5f8f7b16374cd493"} Oct 30 21:46:10.930648 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:10.930611 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc93431044390dcfac81a086e7f15a2acb16df53f24d910d5f8f7b16374cd493" Oct 30 21:46:10.930845 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:10.930831 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" Oct 30 21:46:10.931015 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:10.931003 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" Oct 30 21:46:10.931146 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:10.931137 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" Oct 30 21:46:10.939250 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:10.939228 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" Oct 30 21:46:11.005541 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:10.998907 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" Oct 30 21:46:11.005541 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:10.998947 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" Oct 30 21:46:11.005541 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:10.998980 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" Oct 30 21:46:11.005541 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:10.999009 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" Oct 30 21:46:11.005541 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:10.954639 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" podUID="971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 21:46:11.005541 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:10.954669 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" podUID="0ff8ae26-4364-4c4e-9285-721d1a486a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 21:46:11.005541 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:10.954757 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" podUID="a59a0fc3-8316-47eb-a42e-552d202b1ab7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 30 21:46:11.005541 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.002618 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" Oct 30 21:46:11.005541 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:10.954775 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" podUID="45a5bac0-28ab-4c78-856e-303f6d4522de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": context deadline exceeded" Oct 30 21:46:11.005541 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:10.954791 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" podUID="1e044c02-0053-40e8-9f80-e806cd2d9ae5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 30 21:46:11.005541 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:10.998475 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" Oct 30 21:46:11.005541 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.003842 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" Oct 30 21:46:11.027399 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.017129 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157f1e1f-2nx6l" Oct 30 21:46:11.027399 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.017169 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" event={"ID":"34448c97-8c89-4c69-9e3b-8f513d88b803","Type":"ContainerDied","Data":"9d77c7042cb49ec1e057d977c84e79a1258c7a602a183136226175d01ac3cf2f"} Oct 30 21:46:11.027399 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.017255 4560 scope.go:122] "RemoveContainer" containerID="8944a3b4f6fbac741bd92c01c7ac681a52de24ddde7774811b24688f15708db0" Oct 30 21:46:11.027399 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.019004 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" Oct 30 21:46:11.032357 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.030151 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" Oct 30 21:46:11.032357 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.030628 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" Oct 30 21:46:11.032357 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.032069 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" Oct 30 21:46:11.040115 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.040093 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" Oct 30 21:46:11.057767 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.057737 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cbb50c8-pbs2v"] Oct 30 21:46:11.065181 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.065154 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cbb50c8-pbs2v"] Oct 30 21:46:11.158501 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.158462 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34448c97-8c89-4c69-9e3b-8f513d88b803-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34448c97-8c89-4c69-9e3b-8f513d88b803-istio-data\") pod \"34448c97-8c89-4c69-9e3b-8f513d88b803\" (UID: \"34448c97-8c89-4c69-9e3b-8f513d88b803\") " Oct 30 21:46:11.158702 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.158521 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34448c97-8c89-4c69-9e3b-8f513d88b803-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34448c97-8c89-4c69-9e3b-8f513d88b803-istio-podinfo\") pod \"34448c97-8c89-4c69-9e3b-8f513d88b803\" (UID: \"34448c97-8c89-4c69-9e3b-8f513d88b803\") " Oct 30 21:46:11.158702 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.158557 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34448c97-8c89-4c69-9e3b-8f513d88b803-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34448c97-8c89-4c69-9e3b-8f513d88b803-istio-envoy\") pod \"34448c97-8c89-4c69-9e3b-8f513d88b803\" (UID: \"34448c97-8c89-4c69-9e3b-8f513d88b803\") " Oct 30 21:46:11.158702 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.158586 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34448c97-8c89-4c69-9e3b-8f513d88b803-kube-api-access-fqk7t\" (UniqueName: \"kubernetes.io/projected/34448c97-8c89-4c69-9e3b-8f513d88b803-kube-api-access-fqk7t\") pod \"34448c97-8c89-4c69-9e3b-8f513d88b803\" (UID: \"34448c97-8c89-4c69-9e3b-8f513d88b803\") " Oct 30 21:46:11.158702 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.158675 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34448c97-8c89-4c69-9e3b-8f513d88b803-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34448c97-8c89-4c69-9e3b-8f513d88b803-credential-socket\") pod \"34448c97-8c89-4c69-9e3b-8f513d88b803\" (UID: \"34448c97-8c89-4c69-9e3b-8f513d88b803\") " Oct 30 21:46:11.158910 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.158701 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34448c97-8c89-4c69-9e3b-8f513d88b803-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34448c97-8c89-4c69-9e3b-8f513d88b803-workload-certs\") pod \"34448c97-8c89-4c69-9e3b-8f513d88b803\" (UID: \"34448c97-8c89-4c69-9e3b-8f513d88b803\") " Oct 30 21:46:11.158910 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.158764 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34448c97-8c89-4c69-9e3b-8f513d88b803-istio-data" pod "34448c97-8c89-4c69-9e3b-8f513d88b803" (UID: "34448c97-8c89-4c69-9e3b-8f513d88b803"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:11.158910 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.158863 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34448c97-8c89-4c69-9e3b-8f513d88b803-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34448c97-8c89-4c69-9e3b-8f513d88b803-workload-socket\") pod \"34448c97-8c89-4c69-9e3b-8f513d88b803\" (UID: \"34448c97-8c89-4c69-9e3b-8f513d88b803\") " Oct 30 21:46:11.159060 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.158926 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34448c97-8c89-4c69-9e3b-8f513d88b803-istio-token\" (UniqueName: \"kubernetes.io/projected/34448c97-8c89-4c69-9e3b-8f513d88b803-istio-token\") pod \"34448c97-8c89-4c69-9e3b-8f513d88b803\" (UID: \"34448c97-8c89-4c69-9e3b-8f513d88b803\") " Oct 30 21:46:11.159060 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.158980 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34448c97-8c89-4c69-9e3b-8f513d88b803-credential-socket" pod "34448c97-8c89-4c69-9e3b-8f513d88b803" (UID: "34448c97-8c89-4c69-9e3b-8f513d88b803"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:11.159060 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.159037 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34448c97-8c89-4c69-9e3b-8f513d88b803-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34448c97-8c89-4c69-9e3b-8f513d88b803-istiod-ca-cert\") pod \"34448c97-8c89-4c69-9e3b-8f513d88b803\" (UID: \"34448c97-8c89-4c69-9e3b-8f513d88b803\") " Oct 30 21:46:11.159215 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.159115 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34448c97-8c89-4c69-9e3b-8f513d88b803-workload-certs" pod "34448c97-8c89-4c69-9e3b-8f513d88b803" (UID: "34448c97-8c89-4c69-9e3b-8f513d88b803"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:11.159215 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.159172 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34448c97-8c89-4c69-9e3b-8f513d88b803-workload-socket" pod "34448c97-8c89-4c69-9e3b-8f513d88b803" (UID: "34448c97-8c89-4c69-9e3b-8f513d88b803"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:11.159571 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.159547 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34448c97-8c89-4c69-9e3b-8f513d88b803-istiod-ca-cert" pod "34448c97-8c89-4c69-9e3b-8f513d88b803" (UID: "34448c97-8c89-4c69-9e3b-8f513d88b803"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:11.161696 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.161670 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34448c97-8c89-4c69-9e3b-8f513d88b803-istiod-ca-cert\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:11.161696 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.161694 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34448c97-8c89-4c69-9e3b-8f513d88b803-istio-data\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:11.161884 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.161709 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34448c97-8c89-4c69-9e3b-8f513d88b803-credential-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:11.161884 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.161721 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34448c97-8c89-4c69-9e3b-8f513d88b803-workload-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:11.161884 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.161733 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34448c97-8c89-4c69-9e3b-8f513d88b803-workload-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:11.162165 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.162141 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34448c97-8c89-4c69-9e3b-8f513d88b803-istio-token" pod "34448c97-8c89-4c69-9e3b-8f513d88b803" (UID: "34448c97-8c89-4c69-9e3b-8f513d88b803"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:11.162318 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.162286 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34448c97-8c89-4c69-9e3b-8f513d88b803-istio-envoy" pod "34448c97-8c89-4c69-9e3b-8f513d88b803" (UID: "34448c97-8c89-4c69-9e3b-8f513d88b803"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:11.163483 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.163458 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34448c97-8c89-4c69-9e3b-8f513d88b803-istio-podinfo" pod "34448c97-8c89-4c69-9e3b-8f513d88b803" (UID: "34448c97-8c89-4c69-9e3b-8f513d88b803"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:46:11.164078 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.164031 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34448c97-8c89-4c69-9e3b-8f513d88b803-kube-api-access-fqk7t" pod "34448c97-8c89-4c69-9e3b-8f513d88b803" (UID: "34448c97-8c89-4c69-9e3b-8f513d88b803"). InnerVolumeSpecName "kube-api-access-fqk7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:11.262913 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.262873 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34448c97-8c89-4c69-9e3b-8f513d88b803-istio-token\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:11.262913 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.262905 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34448c97-8c89-4c69-9e3b-8f513d88b803-istio-podinfo\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:11.262913 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.262915 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34448c97-8c89-4c69-9e3b-8f513d88b803-istio-envoy\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:11.262913 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.262925 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqk7t\" (UniqueName: \"kubernetes.io/projected/34448c97-8c89-4c69-9e3b-8f513d88b803-kube-api-access-fqk7t\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:11.418450 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.418377 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f22ef26-vww99"] Oct 30 21:46:11.418893 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.418881 4560 cpu_manager.go:414] "containerMap: removing container" podUID="34448c97-8c89-4c69-9e3b-8f513d88b803" containerName="istio-proxy" Oct 30 21:46:11.418945 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.418895 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34448c97-8c89-4c69-9e3b-8f513d88b803" containerName="istio-proxy" Oct 30 21:46:11.418945 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.418913 4560 cpu_manager.go:414] "containerMap: removing container" podUID="0dbb0932-e12b-4768-861b-4832d25dd952" containerName="revoke-keys" Oct 30 21:46:11.418945 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.418918 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dbb0932-e12b-4768-861b-4832d25dd952" containerName="revoke-keys" Oct 30 21:46:11.419037 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.419008 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dbb0932-e12b-4768-861b-4832d25dd952" containerName="revoke-keys" Oct 30 21:46:11.419037 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.419021 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="34448c97-8c89-4c69-9e3b-8f513d88b803" containerName="istio-proxy" Oct 30 21:46:11.423928 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.423910 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f22ef26-vww99" Oct 30 21:46:11.484777 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.484748 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f22ef26-vww99"] Oct 30 21:46:11.495403 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.495333 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:46:11.496526 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.496471 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:46:11.496699 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.496513 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:46:11.496766 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.496568 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:46:11.591762 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.568384 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2711f3d0-ec20-4184-bc59-0d3a36c6abf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f22ef26-vww99\" (UID: \"2711f3d0-ec20-4184-bc59-0d3a36c6abf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f22ef26-vww99" Oct 30 21:46:11.621436 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.620754 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2711f3d0-ec20-4184-bc59-0d3a36c6abf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f22ef26-vww99\" (UID: \"2711f3d0-ec20-4184-bc59-0d3a36c6abf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f22ef26-vww99" Oct 30 21:46:11.621436 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.621004 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78cxp\" (UniqueName: \"kubernetes.io/projected/2711f3d0-ec20-4184-bc59-0d3a36c6abf3-kube-api-access-78cxp\") pod \"maas-api-revoke-sub-3f22ef26-vww99\" (UID: \"2711f3d0-ec20-4184-bc59-0d3a36c6abf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f22ef26-vww99" Oct 30 21:46:11.750171 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.750130 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2711f3d0-ec20-4184-bc59-0d3a36c6abf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f22ef26-vww99\" (UID: \"2711f3d0-ec20-4184-bc59-0d3a36c6abf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f22ef26-vww99" Oct 30 21:46:11.750489 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.750306 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2711f3d0-ec20-4184-bc59-0d3a36c6abf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f22ef26-vww99\" (UID: \"2711f3d0-ec20-4184-bc59-0d3a36c6abf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f22ef26-vww99" Oct 30 21:46:11.750489 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.750435 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78cxp\" (UniqueName: \"kubernetes.io/projected/2711f3d0-ec20-4184-bc59-0d3a36c6abf3-kube-api-access-78cxp\") pod \"maas-api-revoke-sub-3f22ef26-vww99\" (UID: \"2711f3d0-ec20-4184-bc59-0d3a36c6abf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f22ef26-vww99" Oct 30 21:46:11.751655 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.751559 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2711f3d0-ec20-4184-bc59-0d3a36c6abf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f22ef26-vww99\" (UID: \"2711f3d0-ec20-4184-bc59-0d3a36c6abf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f22ef26-vww99" Oct 30 21:46:11.760729 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.760702 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2711f3d0-ec20-4184-bc59-0d3a36c6abf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f22ef26-vww99\" (UID: \"2711f3d0-ec20-4184-bc59-0d3a36c6abf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f22ef26-vww99" Oct 30 21:46:11.811141 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:11.811116 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78cxp\" (UniqueName: \"kubernetes.io/projected/2711f3d0-ec20-4184-bc59-0d3a36c6abf3-kube-api-access-78cxp\") pod \"maas-api-revoke-sub-3f22ef26-vww99\" (UID: \"2711f3d0-ec20-4184-bc59-0d3a36c6abf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f22ef26-vww99" Oct 30 21:46:12.028117 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:12.028024 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw" Oct 30 21:46:12.059124 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:12.059101 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f22ef26-vww99" Oct 30 21:46:12.464787 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:12.464709 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f22ef26-vww99"] Oct 30 21:46:12.469570 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:46:12.469532 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2711f3d0_ec20_4184_bc59_0d3a36c6abf3.slice/crio-37e2b2dcdc34f3ec1219fb98b282048c954d7829ca8156b3e31cd46f89257cc5 WatchSource:0}: Error finding container 37e2b2dcdc34f3ec1219fb98b282048c954d7829ca8156b3e31cd46f89257cc5: Status 404 returned error can't find the container with id 37e2b2dcdc34f3ec1219fb98b282048c954d7829ca8156b3e31cd46f89257cc5 Oct 30 21:46:12.579216 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:12.579180 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b0280a3-57f9-4fdd-9e32-3c796db6f2d1" path="/var/lib/kubelet/pods/5b0280a3-57f9-4fdd-9e32-3c796db6f2d1/volumes" Oct 30 21:46:13.037854 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:13.037772 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f22ef26-vww99" event={"ID":"2711f3d0-ec20-4184-bc59-0d3a36c6abf3","Type":"ContainerStarted","Data":"e913f3bcdba072fd1d0bc6f9dfbabddbe8ae0346cc6c35ee68b41d4888ea81e6"} Oct 30 21:46:13.037993 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:13.037857 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f22ef26-vww99" event={"ID":"2711f3d0-ec20-4184-bc59-0d3a36c6abf3","Type":"ContainerStarted","Data":"37e2b2dcdc34f3ec1219fb98b282048c954d7829ca8156b3e31cd46f89257cc5"} Oct 30 21:46:13.204792 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:13.204744 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f22ef26-vww99" podStartSLOduration=2.204729842 podStartE2EDuration="2.204729842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:13.202179209 +0000 UTC m=+1948.756878984" watchObservedRunningTime="2026-10-30 21:46:13.204729842 +0000 UTC m=+1948.759429675" Oct 30 21:46:13.312149 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:13.312027 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw"] Oct 30 21:46:13.315916 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:13.315888 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pcphw"] Oct 30 21:46:14.046747 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:14.046716 4560 generic.go:356] "Generic (PLEG): container finished" podID="2711f3d0-ec20-4184-bc59-0d3a36c6abf3" containerID="e913f3bcdba072fd1d0bc6f9dfbabddbe8ae0346cc6c35ee68b41d4888ea81e6" exitCode=0 Oct 30 21:46:14.046991 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:14.046788 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f22ef26-vww99" event={"ID":"2711f3d0-ec20-4184-bc59-0d3a36c6abf3","Type":"ContainerDied","Data":"e913f3bcdba072fd1d0bc6f9dfbabddbe8ae0346cc6c35ee68b41d4888ea81e6"} Oct 30 21:46:14.577083 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:14.577042 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34448c97-8c89-4c69-9e3b-8f513d88b803" path="/var/lib/kubelet/pods/34448c97-8c89-4c69-9e3b-8f513d88b803/volumes" Oct 30 21:46:15.217959 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:15.217935 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f22ef26-vww99" Oct 30 21:46:15.303774 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:15.303702 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2711f3d0-ec20-4184-bc59-0d3a36c6abf3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2711f3d0-ec20-4184-bc59-0d3a36c6abf3-maas-api-openshift-ca\") pod \"2711f3d0-ec20-4184-bc59-0d3a36c6abf3\" (UID: \"2711f3d0-ec20-4184-bc59-0d3a36c6abf3\") " Oct 30 21:46:15.303774 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:15.303753 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2711f3d0-ec20-4184-bc59-0d3a36c6abf3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2711f3d0-ec20-4184-bc59-0d3a36c6abf3-maas-api-xks-ca\") pod \"2711f3d0-ec20-4184-bc59-0d3a36c6abf3\" (UID: \"2711f3d0-ec20-4184-bc59-0d3a36c6abf3\") " Oct 30 21:46:15.303934 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:15.303814 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2711f3d0-ec20-4184-bc59-0d3a36c6abf3-kube-api-access-78cxp\" (UniqueName: \"kubernetes.io/projected/2711f3d0-ec20-4184-bc59-0d3a36c6abf3-kube-api-access-78cxp\") pod \"2711f3d0-ec20-4184-bc59-0d3a36c6abf3\" (UID: \"2711f3d0-ec20-4184-bc59-0d3a36c6abf3\") " Oct 30 21:46:15.304150 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:15.304122 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2711f3d0-ec20-4184-bc59-0d3a36c6abf3-maas-api-openshift-ca" pod "2711f3d0-ec20-4184-bc59-0d3a36c6abf3" (UID: "2711f3d0-ec20-4184-bc59-0d3a36c6abf3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:15.304421 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:15.304401 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2711f3d0-ec20-4184-bc59-0d3a36c6abf3-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:15.305956 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:15.305927 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2711f3d0-ec20-4184-bc59-0d3a36c6abf3-kube-api-access-78cxp" pod "2711f3d0-ec20-4184-bc59-0d3a36c6abf3" (UID: "2711f3d0-ec20-4184-bc59-0d3a36c6abf3"). InnerVolumeSpecName "kube-api-access-78cxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:15.305956 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:15.305937 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2711f3d0-ec20-4184-bc59-0d3a36c6abf3-maas-api-xks-ca" pod "2711f3d0-ec20-4184-bc59-0d3a36c6abf3" (UID: "2711f3d0-ec20-4184-bc59-0d3a36c6abf3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:15.405738 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:15.405706 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2711f3d0-ec20-4184-bc59-0d3a36c6abf3-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:15.405738 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:15.405733 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78cxp\" (UniqueName: \"kubernetes.io/projected/2711f3d0-ec20-4184-bc59-0d3a36c6abf3-kube-api-access-78cxp\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:16.092300 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:16.092091 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f22ef26-vww99" Oct 30 21:46:16.092300 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:16.091954 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f22ef26-vww99" event={"ID":"2711f3d0-ec20-4184-bc59-0d3a36c6abf3","Type":"ContainerDied","Data":"37e2b2dcdc34f3ec1219fb98b282048c954d7829ca8156b3e31cd46f89257cc5"} Oct 30 21:46:16.092300 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:16.092242 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37e2b2dcdc34f3ec1219fb98b282048c954d7829ca8156b3e31cd46f89257cc5" Oct 30 21:46:17.631649 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:17.631581 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg"] Oct 30 21:46:17.632501 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:17.632464 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" podUID="eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9" containerName="istio-proxy" containerID="cri-o://cb461b982e91e25e228af1a6e6bdbfc04b08a2860948353f8379df2c87c524aa" gracePeriod=30 Oct 30 21:46:19.685774 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:46:19.685741 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:46:22.167690 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.167660 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e24f13ff-z9rch"] Oct 30 21:46:22.168193 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.168180 4560 cpu_manager.go:414] "containerMap: removing container" podUID="2711f3d0-ec20-4184-bc59-0d3a36c6abf3" containerName="revoke-keys" Oct 30 21:46:22.168240 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.168194 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2711f3d0-ec20-4184-bc59-0d3a36c6abf3" containerName="revoke-keys" Oct 30 21:46:22.168282 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.168272 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="2711f3d0-ec20-4184-bc59-0d3a36c6abf3" containerName="revoke-keys" Oct 30 21:46:22.172597 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.172581 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24f13ff-z9rch" Oct 30 21:46:22.228495 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.228470 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e24f13ff-z9rch"] Oct 30 21:46:22.237149 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.237120 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e22781aa-a00f-4608-956a-a5c81d03641b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e24f13ff-z9rch\" (UID: \"e22781aa-a00f-4608-956a-a5c81d03641b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24f13ff-z9rch" Oct 30 21:46:22.237280 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.237163 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e22781aa-a00f-4608-956a-a5c81d03641b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e24f13ff-z9rch\" (UID: \"e22781aa-a00f-4608-956a-a5c81d03641b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24f13ff-z9rch" Oct 30 21:46:22.237399 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.237372 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2572q\" (UniqueName: \"kubernetes.io/projected/e22781aa-a00f-4608-956a-a5c81d03641b-kube-api-access-2572q\") pod \"maas-api-revoke-sub-e24f13ff-z9rch\" (UID: \"e22781aa-a00f-4608-956a-a5c81d03641b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24f13ff-z9rch" Oct 30 21:46:22.270663 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.270641 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:46:22.270763 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.270690 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:46:22.280438 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.280419 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:46:22.280438 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.280430 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:46:22.338840 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.338821 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e22781aa-a00f-4608-956a-a5c81d03641b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e24f13ff-z9rch\" (UID: \"e22781aa-a00f-4608-956a-a5c81d03641b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24f13ff-z9rch" Oct 30 21:46:22.338943 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.338849 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e22781aa-a00f-4608-956a-a5c81d03641b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e24f13ff-z9rch\" (UID: \"e22781aa-a00f-4608-956a-a5c81d03641b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24f13ff-z9rch" Oct 30 21:46:22.338943 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.338885 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2572q\" (UniqueName: \"kubernetes.io/projected/e22781aa-a00f-4608-956a-a5c81d03641b-kube-api-access-2572q\") pod \"maas-api-revoke-sub-e24f13ff-z9rch\" (UID: \"e22781aa-a00f-4608-956a-a5c81d03641b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24f13ff-z9rch" Oct 30 21:46:22.339452 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.339432 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e22781aa-a00f-4608-956a-a5c81d03641b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e24f13ff-z9rch\" (UID: \"e22781aa-a00f-4608-956a-a5c81d03641b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24f13ff-z9rch" Oct 30 21:46:22.341216 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.341192 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e22781aa-a00f-4608-956a-a5c81d03641b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e24f13ff-z9rch\" (UID: \"e22781aa-a00f-4608-956a-a5c81d03641b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24f13ff-z9rch" Oct 30 21:46:22.397422 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.397399 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2572q\" (UniqueName: \"kubernetes.io/projected/e22781aa-a00f-4608-956a-a5c81d03641b-kube-api-access-2572q\") pod \"maas-api-revoke-sub-e24f13ff-z9rch\" (UID: \"e22781aa-a00f-4608-956a-a5c81d03641b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24f13ff-z9rch" Oct 30 21:46:22.485467 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.485440 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24f13ff-z9rch" Oct 30 21:46:22.868798 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.868772 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e24f13ff-z9rch"] Oct 30 21:46:22.870586 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:46:22.870553 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode22781aa_a00f_4608_956a_a5c81d03641b.slice/crio-a5668a7ff1ec691c2dea66cd3982e508f580bbe83fcb622c989aad2b4c42d828 WatchSource:0}: Error finding container a5668a7ff1ec691c2dea66cd3982e508f580bbe83fcb622c989aad2b4c42d828: Status 404 returned error can't find the container with id a5668a7ff1ec691c2dea66cd3982e508f580bbe83fcb622c989aad2b4c42d828 Oct 30 21:46:22.885229 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.885208 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" Oct 30 21:46:22.948731 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.948707 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-workload-certs\") pod \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\" (UID: \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\") " Oct 30 21:46:22.948828 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.948808 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-istio-data\") pod \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\" (UID: \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\") " Oct 30 21:46:22.948876 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.948851 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-credential-socket\") pod \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\" (UID: \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\") " Oct 30 21:46:22.948914 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.948900 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-kube-api-access-xxw95\" (UniqueName: \"kubernetes.io/projected/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-kube-api-access-xxw95\") pod \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\" (UID: \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\") " Oct 30 21:46:22.948964 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.948938 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-workload-certs" pod "eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9" (UID: "eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:22.948964 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.948946 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-istio-podinfo\") pod \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\" (UID: \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\") " Oct 30 21:46:22.949066 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.949015 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-istiod-ca-cert\") pod \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\" (UID: \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\") " Oct 30 21:46:22.949118 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.949078 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-istio-token\" (UniqueName: \"kubernetes.io/projected/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-istio-token\") pod \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\" (UID: \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\") " Oct 30 21:46:22.949118 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.949092 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-istio-data" pod "eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9" (UID: "eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:22.949226 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.949190 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-istio-envoy\") pod \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\" (UID: \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\") " Oct 30 21:46:22.949226 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.949200 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-credential-socket" pod "eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9" (UID: "eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:22.949226 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.949223 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-workload-socket\") pod \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\" (UID: \"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9\") " Oct 30 21:46:22.949513 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.949490 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-workload-socket" pod "eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9" (UID: "eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:22.949619 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.949435 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-istiod-ca-cert" pod "eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9" (UID: "eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:22.949857 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.949836 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-istio-data\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:22.949909 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.949863 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-credential-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:22.949909 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.949880 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-istiod-ca-cert\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:22.949909 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.949894 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-workload-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:22.949996 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.949908 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-workload-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:22.951050 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.951029 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-istio-podinfo" pod "eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9" (UID: "eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:46:22.951185 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.951168 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-istio-envoy" pod "eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9" (UID: "eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:22.951406 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.951387 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-kube-api-access-xxw95" pod "eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9" (UID: "eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9"). InnerVolumeSpecName "kube-api-access-xxw95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:22.951466 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:22.951439 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-istio-token" pod "eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9" (UID: "eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:23.060118 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:23.060081 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-istio-envoy\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:23.060270 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:23.060121 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxw95\" (UniqueName: \"kubernetes.io/projected/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-kube-api-access-xxw95\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:23.060270 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:23.060143 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-istio-podinfo\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:23.060270 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:23.060157 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9-istio-token\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:23.168464 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:23.168386 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24f13ff-z9rch" event={"ID":"e22781aa-a00f-4608-956a-a5c81d03641b","Type":"ContainerStarted","Data":"a5668a7ff1ec691c2dea66cd3982e508f580bbe83fcb622c989aad2b4c42d828"} Oct 30 21:46:23.170028 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:23.169996 4560 generic.go:356] "Generic (PLEG): container finished" podID="eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9" containerID="cb461b982e91e25e228af1a6e6bdbfc04b08a2860948353f8379df2c87c524aa" exitCode=0 Oct 30 21:46:23.170163 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:23.170032 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" event={"ID":"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9","Type":"ContainerDied","Data":"cb461b982e91e25e228af1a6e6bdbfc04b08a2860948353f8379df2c87c524aa"} Oct 30 21:46:23.170163 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:23.170067 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" Oct 30 21:46:23.170163 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:23.170081 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg" event={"ID":"eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9","Type":"ContainerDied","Data":"61c2aec7212625da0613a4dca309ebae1c6fd7ff6567c25eb2e67bc9d603f422"} Oct 30 21:46:23.170163 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:23.170108 4560 scope.go:122] "RemoveContainer" containerID="cb461b982e91e25e228af1a6e6bdbfc04b08a2860948353f8379df2c87c524aa" Oct 30 21:46:23.181289 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:23.181272 4560 scope.go:122] "RemoveContainer" containerID="cb461b982e91e25e228af1a6e6bdbfc04b08a2860948353f8379df2c87c524aa" Oct 30 21:46:23.181582 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:46:23.181563 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb461b982e91e25e228af1a6e6bdbfc04b08a2860948353f8379df2c87c524aa\": container with ID starting with cb461b982e91e25e228af1a6e6bdbfc04b08a2860948353f8379df2c87c524aa not found: ID does not exist" containerID="cb461b982e91e25e228af1a6e6bdbfc04b08a2860948353f8379df2c87c524aa" Oct 30 21:46:23.181658 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:23.181587 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb461b982e91e25e228af1a6e6bdbfc04b08a2860948353f8379df2c87c524aa"} err="failed to get container status \"cb461b982e91e25e228af1a6e6bdbfc04b08a2860948353f8379df2c87c524aa\": rpc error: code = NotFound desc = could not find container \"cb461b982e91e25e228af1a6e6bdbfc04b08a2860948353f8379df2c87c524aa\": container with ID starting with cb461b982e91e25e228af1a6e6bdbfc04b08a2860948353f8379df2c87c524aa not found: ID does not exist" Oct 30 21:46:23.333291 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:23.333186 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg"] Oct 30 21:46:23.337252 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:23.337213 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ssqzg"] Oct 30 21:46:24.179812 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:24.179737 4560 generic.go:356] "Generic (PLEG): container finished" podID="e22781aa-a00f-4608-956a-a5c81d03641b" containerID="078dc5a1eb234899b375487b465286c600b4928df9e608954ecc29067dc06c2e" exitCode=0 Oct 30 21:46:24.180290 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:24.179825 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24f13ff-z9rch" event={"ID":"e22781aa-a00f-4608-956a-a5c81d03641b","Type":"ContainerDied","Data":"078dc5a1eb234899b375487b465286c600b4928df9e608954ecc29067dc06c2e"} Oct 30 21:46:24.576498 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:24.576469 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9" path="/var/lib/kubelet/pods/eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9/volumes" Oct 30 21:46:25.373429 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:25.373397 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24f13ff-z9rch" Oct 30 21:46:25.432573 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:25.432536 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e22781aa-a00f-4608-956a-a5c81d03641b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e22781aa-a00f-4608-956a-a5c81d03641b-maas-api-openshift-ca\") pod \"e22781aa-a00f-4608-956a-a5c81d03641b\" (UID: \"e22781aa-a00f-4608-956a-a5c81d03641b\") " Oct 30 21:46:25.432762 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:25.432594 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e22781aa-a00f-4608-956a-a5c81d03641b-kube-api-access-2572q\" (UniqueName: \"kubernetes.io/projected/e22781aa-a00f-4608-956a-a5c81d03641b-kube-api-access-2572q\") pod \"e22781aa-a00f-4608-956a-a5c81d03641b\" (UID: \"e22781aa-a00f-4608-956a-a5c81d03641b\") " Oct 30 21:46:25.432762 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:25.432707 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e22781aa-a00f-4608-956a-a5c81d03641b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e22781aa-a00f-4608-956a-a5c81d03641b-maas-api-xks-ca\") pod \"e22781aa-a00f-4608-956a-a5c81d03641b\" (UID: \"e22781aa-a00f-4608-956a-a5c81d03641b\") " Oct 30 21:46:25.433002 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:25.432975 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e22781aa-a00f-4608-956a-a5c81d03641b-maas-api-openshift-ca" pod "e22781aa-a00f-4608-956a-a5c81d03641b" (UID: "e22781aa-a00f-4608-956a-a5c81d03641b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:25.433317 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:25.433300 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e22781aa-a00f-4608-956a-a5c81d03641b-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:25.434770 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:25.434746 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e22781aa-a00f-4608-956a-a5c81d03641b-maas-api-xks-ca" pod "e22781aa-a00f-4608-956a-a5c81d03641b" (UID: "e22781aa-a00f-4608-956a-a5c81d03641b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:25.434770 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:25.434757 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e22781aa-a00f-4608-956a-a5c81d03641b-kube-api-access-2572q" pod "e22781aa-a00f-4608-956a-a5c81d03641b" (UID: "e22781aa-a00f-4608-956a-a5c81d03641b"). InnerVolumeSpecName "kube-api-access-2572q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:25.534290 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:25.534256 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2572q\" (UniqueName: \"kubernetes.io/projected/e22781aa-a00f-4608-956a-a5c81d03641b-kube-api-access-2572q\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:25.534290 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:25.534286 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e22781aa-a00f-4608-956a-a5c81d03641b-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:26.233359 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:26.233316 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24f13ff-z9rch" Oct 30 21:46:26.233689 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:26.233658 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24f13ff-z9rch" event={"ID":"e22781aa-a00f-4608-956a-a5c81d03641b","Type":"ContainerDied","Data":"a5668a7ff1ec691c2dea66cd3982e508f580bbe83fcb622c989aad2b4c42d828"} Oct 30 21:46:26.233834 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:26.233814 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5668a7ff1ec691c2dea66cd3982e508f580bbe83fcb622c989aad2b4c42d828" Oct 30 21:46:33.522266 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:33.522221 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh"] Oct 30 21:46:33.522769 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:33.522753 4560 cpu_manager.go:414] "containerMap: removing container" podUID="eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9" containerName="istio-proxy" Oct 30 21:46:33.522814 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:33.522770 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9" containerName="istio-proxy" Oct 30 21:46:33.522814 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:33.522781 4560 cpu_manager.go:414] "containerMap: removing container" podUID="e22781aa-a00f-4608-956a-a5c81d03641b" containerName="revoke-keys" Oct 30 21:46:33.522814 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:33.522787 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e22781aa-a00f-4608-956a-a5c81d03641b" containerName="revoke-keys" Oct 30 21:46:33.522911 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:33.522864 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="eda3d2f4-4c1d-4d24-8fd8-96ba3dbc93c9" containerName="istio-proxy" Oct 30 21:46:33.522911 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:33.522871 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="e22781aa-a00f-4608-956a-a5c81d03641b" containerName="revoke-keys" Oct 30 21:46:33.527079 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:33.527061 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" Oct 30 21:46:33.579126 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:33.579102 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-19dd4f-openshift-default-dockercfg-rbrph\"" Oct 30 21:46:33.630720 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:33.630693 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh"] Oct 30 21:46:33.653897 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:33.653873 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bfd4837-aa59-491c-8e32-770bf1451706-istio-podinfo\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh\" (UID: \"1bfd4837-aa59-491c-8e32-770bf1451706\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" Oct 30 21:46:33.654018 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:33.653914 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bfd4837-aa59-491c-8e32-770bf1451706-istio-envoy\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh\" (UID: \"1bfd4837-aa59-491c-8e32-770bf1451706\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" Oct 30 21:46:33.654018 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:33.653994 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfd4837-aa59-491c-8e32-770bf1451706-workload-socket\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh\" (UID: \"1bfd4837-aa59-491c-8e32-770bf1451706\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" Oct 30 21:46:33.654150 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:33.654132 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bfd4837-aa59-491c-8e32-770bf1451706-istio-data\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh\" (UID: \"1bfd4837-aa59-491c-8e32-770bf1451706\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" Oct 30 21:46:33.654186 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:33.654158 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bfd4837-aa59-491c-8e32-770bf1451706-istio-token\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh\" (UID: \"1bfd4837-aa59-491c-8e32-770bf1451706\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" Oct 30 21:46:33.654186 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:33.654175 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bfd4837-aa59-491c-8e32-770bf1451706-istiod-ca-cert\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh\" (UID: \"1bfd4837-aa59-491c-8e32-770bf1451706\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" Oct 30 21:46:33.654314 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:33.654302 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bfd4837-aa59-491c-8e32-770bf1451706-workload-certs\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh\" (UID: \"1bfd4837-aa59-491c-8e32-770bf1451706\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" Oct 30 21:46:33.654388 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:33.654372 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfd4837-aa59-491c-8e32-770bf1451706-credential-socket\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh\" (UID: \"1bfd4837-aa59-491c-8e32-770bf1451706\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" Oct 30 21:46:33.654421 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:33.654410 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxgrn\" (UniqueName: \"kubernetes.io/projected/1bfd4837-aa59-491c-8e32-770bf1451706-kube-api-access-bxgrn\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh\" (UID: \"1bfd4837-aa59-491c-8e32-770bf1451706\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" Oct 30 21:46:33.756545 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:33.756522 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bfd4837-aa59-491c-8e32-770bf1451706-istio-data\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh\" (UID: \"1bfd4837-aa59-491c-8e32-770bf1451706\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" Oct 30 21:46:33.756688 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:33.756550 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bfd4837-aa59-491c-8e32-770bf1451706-istio-token\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh\" (UID: \"1bfd4837-aa59-491c-8e32-770bf1451706\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" Oct 30 21:46:33.756688 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:33.756568 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bfd4837-aa59-491c-8e32-770bf1451706-istiod-ca-cert\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh\" (UID: \"1bfd4837-aa59-491c-8e32-770bf1451706\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" Oct 30 21:46:33.756688 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:33.756624 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bfd4837-aa59-491c-8e32-770bf1451706-workload-certs\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh\" (UID: \"1bfd4837-aa59-491c-8e32-770bf1451706\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" Oct 30 21:46:33.756688 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:33.756671 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfd4837-aa59-491c-8e32-770bf1451706-credential-socket\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh\" (UID: \"1bfd4837-aa59-491c-8e32-770bf1451706\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" Oct 30 21:46:33.756903 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:33.756701 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxgrn\" (UniqueName: \"kubernetes.io/projected/1bfd4837-aa59-491c-8e32-770bf1451706-kube-api-access-bxgrn\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh\" (UID: \"1bfd4837-aa59-491c-8e32-770bf1451706\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" Oct 30 21:46:33.756903 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:33.756728 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bfd4837-aa59-491c-8e32-770bf1451706-istio-podinfo\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh\" (UID: \"1bfd4837-aa59-491c-8e32-770bf1451706\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" Oct 30 21:46:33.757007 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:33.756934 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bfd4837-aa59-491c-8e32-770bf1451706-istio-envoy\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh\" (UID: \"1bfd4837-aa59-491c-8e32-770bf1451706\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" Oct 30 21:46:33.757064 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:33.757016 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bfd4837-aa59-491c-8e32-770bf1451706-istio-data\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh\" (UID: \"1bfd4837-aa59-491c-8e32-770bf1451706\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" Oct 30 21:46:33.757121 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:33.757066 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bfd4837-aa59-491c-8e32-770bf1451706-workload-certs\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh\" (UID: \"1bfd4837-aa59-491c-8e32-770bf1451706\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" Oct 30 21:46:33.757178 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:33.757155 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfd4837-aa59-491c-8e32-770bf1451706-credential-socket\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh\" (UID: \"1bfd4837-aa59-491c-8e32-770bf1451706\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" Oct 30 21:46:33.757370 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:33.757311 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfd4837-aa59-491c-8e32-770bf1451706-workload-socket\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh\" (UID: \"1bfd4837-aa59-491c-8e32-770bf1451706\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" Oct 30 21:46:33.757469 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:33.757371 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bfd4837-aa59-491c-8e32-770bf1451706-istiod-ca-cert\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh\" (UID: \"1bfd4837-aa59-491c-8e32-770bf1451706\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" Oct 30 21:46:33.757635 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:33.757613 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfd4837-aa59-491c-8e32-770bf1451706-workload-socket\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh\" (UID: \"1bfd4837-aa59-491c-8e32-770bf1451706\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" Oct 30 21:46:33.759367 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:33.759315 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bfd4837-aa59-491c-8e32-770bf1451706-istio-podinfo\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh\" (UID: \"1bfd4837-aa59-491c-8e32-770bf1451706\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" Oct 30 21:46:33.759367 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:33.759350 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bfd4837-aa59-491c-8e32-770bf1451706-istio-envoy\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh\" (UID: \"1bfd4837-aa59-491c-8e32-770bf1451706\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" Oct 30 21:46:33.815691 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:33.815632 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bfd4837-aa59-491c-8e32-770bf1451706-istio-token\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh\" (UID: \"1bfd4837-aa59-491c-8e32-770bf1451706\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" Oct 30 21:46:33.815997 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:33.815979 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxgrn\" (UniqueName: \"kubernetes.io/projected/1bfd4837-aa59-491c-8e32-770bf1451706-kube-api-access-bxgrn\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh\" (UID: \"1bfd4837-aa59-491c-8e32-770bf1451706\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" Oct 30 21:46:33.841089 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:33.841069 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" Oct 30 21:46:34.300221 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:34.300184 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh"] Oct 30 21:46:34.305666 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:46:34.305629 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bfd4837_aa59_491c_8e32_770bf1451706.slice/crio-0c31d18d802f93990cee3c89a94222737bba0499be114dff11e325fe6227add4 WatchSource:0}: Error finding container 0c31d18d802f93990cee3c89a94222737bba0499be114dff11e325fe6227add4: Status 404 returned error can't find the container with id 0c31d18d802f93990cee3c89a94222737bba0499be114dff11e325fe6227add4 Oct 30 21:46:34.307798 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:34.307765 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:46:34.307871 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:34.307828 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:46:34.307871 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:34.307855 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:46:35.299930 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:35.299890 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" event={"ID":"1bfd4837-aa59-491c-8e32-770bf1451706","Type":"ContainerStarted","Data":"3ab8eb4bd46329cad45ed1a5fd4f76925d0a8e00415497fa1667253f0ec8e7f9"} Oct 30 21:46:35.300373 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:35.299960 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" event={"ID":"1bfd4837-aa59-491c-8e32-770bf1451706","Type":"ContainerStarted","Data":"0c31d18d802f93990cee3c89a94222737bba0499be114dff11e325fe6227add4"} Oct 30 21:46:35.506895 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:35.505200 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" podStartSLOduration=2.505180477 podStartE2EDuration="2.505180477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:35.474390695 +0000 UTC m=+1971.029090473" watchObservedRunningTime="2026-10-30 21:46:35.505180477 +0000 UTC m=+1971.059880257" Oct 30 21:46:35.792521 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:35.792488 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b78ca65b-hbdf2"] Oct 30 21:46:35.796561 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:35.796538 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78ca65b-hbdf2" Oct 30 21:46:35.851647 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:35.851610 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:46:35.851821 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:35.851743 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:46:35.852661 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:35.852589 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" Oct 30 21:46:35.854492 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:35.854318 4560 patch_prober.go:29] interesting pod/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 30 21:46:35.854635 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:35.854545 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" podUID="1bfd4837-aa59-491c-8e32-770bf1451706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 30 21:46:35.856464 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:35.856442 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b78ca65b-hbdf2"] Oct 30 21:46:35.897827 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:35.897794 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:46:35.897827 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:35.897801 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:46:35.900187 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:35.900155 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd250b4b-880a-47b1-bd3c-7843bc9d0f28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b78ca65b-hbdf2\" (UID: \"fd250b4b-880a-47b1-bd3c-7843bc9d0f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78ca65b-hbdf2" Oct 30 21:46:35.900350 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:35.900317 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd250b4b-880a-47b1-bd3c-7843bc9d0f28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b78ca65b-hbdf2\" (UID: \"fd250b4b-880a-47b1-bd3c-7843bc9d0f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78ca65b-hbdf2" Oct 30 21:46:35.900453 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:35.900432 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx7wp\" (UniqueName: \"kubernetes.io/projected/fd250b4b-880a-47b1-bd3c-7843bc9d0f28-kube-api-access-gx7wp\") pod \"maas-api-revoke-sub-b78ca65b-hbdf2\" (UID: \"fd250b4b-880a-47b1-bd3c-7843bc9d0f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78ca65b-hbdf2" Oct 30 21:46:36.001319 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:36.001274 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd250b4b-880a-47b1-bd3c-7843bc9d0f28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b78ca65b-hbdf2\" (UID: \"fd250b4b-880a-47b1-bd3c-7843bc9d0f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78ca65b-hbdf2" Oct 30 21:46:36.001499 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:36.001365 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx7wp\" (UniqueName: \"kubernetes.io/projected/fd250b4b-880a-47b1-bd3c-7843bc9d0f28-kube-api-access-gx7wp\") pod \"maas-api-revoke-sub-b78ca65b-hbdf2\" (UID: \"fd250b4b-880a-47b1-bd3c-7843bc9d0f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78ca65b-hbdf2" Oct 30 21:46:36.001599 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:36.001573 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd250b4b-880a-47b1-bd3c-7843bc9d0f28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b78ca65b-hbdf2\" (UID: \"fd250b4b-880a-47b1-bd3c-7843bc9d0f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78ca65b-hbdf2" Oct 30 21:46:36.002260 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:36.002237 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd250b4b-880a-47b1-bd3c-7843bc9d0f28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b78ca65b-hbdf2\" (UID: \"fd250b4b-880a-47b1-bd3c-7843bc9d0f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78ca65b-hbdf2" Oct 30 21:46:36.019954 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:36.019929 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd250b4b-880a-47b1-bd3c-7843bc9d0f28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b78ca65b-hbdf2\" (UID: \"fd250b4b-880a-47b1-bd3c-7843bc9d0f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78ca65b-hbdf2" Oct 30 21:46:36.071272 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:36.071207 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx7wp\" (UniqueName: \"kubernetes.io/projected/fd250b4b-880a-47b1-bd3c-7843bc9d0f28-kube-api-access-gx7wp\") pod \"maas-api-revoke-sub-b78ca65b-hbdf2\" (UID: \"fd250b4b-880a-47b1-bd3c-7843bc9d0f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78ca65b-hbdf2" Oct 30 21:46:36.130437 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:36.121709 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78ca65b-hbdf2" Oct 30 21:46:36.849274 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:36.847489 4560 patch_prober.go:29] interesting pod/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 30 21:46:36.849274 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:36.847555 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" podUID="1bfd4837-aa59-491c-8e32-770bf1451706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 30 21:46:37.132478 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:37.127844 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b78ca65b-hbdf2"] Oct 30 21:46:37.640106 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:37.639885 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78ca65b-hbdf2" event={"ID":"fd250b4b-880a-47b1-bd3c-7843bc9d0f28","Type":"ContainerStarted","Data":"39cf9086bc38e8834e6a7d76eefa5a749ff96b5e733ccd57f9c620d9f68ad1c7"} Oct 30 21:46:37.842840 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:37.842803 4560 patch_prober.go:29] interesting pod/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 30 21:46:37.843162 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:37.843116 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" podUID="1bfd4837-aa59-491c-8e32-770bf1451706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 30 21:46:38.150446 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:38.149746 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd1f2444-7ss99"] Oct 30 21:46:38.294521 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:38.209843 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1f2444-7ss99" Oct 30 21:46:38.294521 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:38.247483 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd1f2444-7ss99"] Oct 30 21:46:38.403461 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:38.396300 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd1f2444-7ss99\" (UID: \"a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1f2444-7ss99" Oct 30 21:46:38.403461 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:38.396439 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-974h2\" (UniqueName: \"kubernetes.io/projected/a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806-kube-api-access-974h2\") pod \"maas-api-revoke-sub-dd1f2444-7ss99\" (UID: \"a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1f2444-7ss99" Oct 30 21:46:38.403461 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:38.396528 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd1f2444-7ss99\" (UID: \"a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1f2444-7ss99" Oct 30 21:46:38.503921 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:38.499206 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-974h2\" (UniqueName: \"kubernetes.io/projected/a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806-kube-api-access-974h2\") pod \"maas-api-revoke-sub-dd1f2444-7ss99\" (UID: \"a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1f2444-7ss99" Oct 30 21:46:38.503921 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:38.499332 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd1f2444-7ss99\" (UID: \"a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1f2444-7ss99" Oct 30 21:46:38.503921 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:38.499478 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd1f2444-7ss99\" (UID: \"a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1f2444-7ss99" Oct 30 21:46:38.503921 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:38.500379 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd1f2444-7ss99\" (UID: \"a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1f2444-7ss99" Oct 30 21:46:38.587412 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:38.575140 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd1f2444-7ss99\" (UID: \"a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1f2444-7ss99" Oct 30 21:46:38.618365 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:38.596568 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-974h2\" (UniqueName: \"kubernetes.io/projected/a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806-kube-api-access-974h2\") pod \"maas-api-revoke-sub-dd1f2444-7ss99\" (UID: \"a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1f2444-7ss99" Oct 30 21:46:38.726887 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:38.726855 4560 generic.go:356] "Generic (PLEG): container finished" podID="fd250b4b-880a-47b1-bd3c-7843bc9d0f28" containerID="d5577bf78c3f4b709df4f71e206343cd942be78abf666c9d116d9f4df3cae41d" exitCode=0 Oct 30 21:46:38.727193 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:38.727172 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78ca65b-hbdf2" event={"ID":"fd250b4b-880a-47b1-bd3c-7843bc9d0f28","Type":"ContainerDied","Data":"d5577bf78c3f4b709df4f71e206343cd942be78abf666c9d116d9f4df3cae41d"} Oct 30 21:46:38.926821 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:38.926782 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1f2444-7ss99" Oct 30 21:46:38.928183 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:38.928154 4560 patch_prober.go:29] interesting pod/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 30 21:46:38.928474 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:38.928449 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" podUID="1bfd4837-aa59-491c-8e32-770bf1451706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 30 21:46:39.871182 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:39.798522 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" podUID="b80b0fba-3830-4a74-a724-aead35443b89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 30 21:46:39.871182 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:39.858686 4560 patch_prober.go:29] interesting pod/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 30 21:46:39.871182 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:39.858740 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" podUID="1bfd4837-aa59-491c-8e32-770bf1451706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 30 21:46:39.987768 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:39.985473 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" podUID="0155a02e-8df7-4847-b40f-b8d9e4bac8c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:40.019003 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:40.016807 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" podUID="0ff8ae26-4364-4c4e-9285-721d1a486a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:40.019003 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:40.017146 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" podUID="9e2d7535-075d-4211-b814-35d6abff7605" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:46:40.019003 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:40.017382 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" podUID="27df5dfe-47a4-4ebb-bd09-1ceef406600a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded" Oct 30 21:46:40.019003 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:40.018398 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" podUID="a59a0fc3-8316-47eb-a42e-552d202b1ab7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:40.209361 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:40.196601 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" podUID="45a5bac0-28ab-4c78-856e-303f6d4522de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:40.304577 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:40.304437 4560 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" podUID="a59a0fc3-8316-47eb-a42e-552d202b1ab7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:40.312522 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:40.312439 4560 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" podUID="90bfb1e4-9526-40f6-aed4-ee46c2d86471" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:40.324370 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:40.314558 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd1f2444-7ss99"] Oct 30 21:46:40.324677 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:40.324646 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" podUID="8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:40.491381 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:40.486516 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" podUID="e974bae1-bb18-46b0-a4ee-4c01d6455e32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:40.532053 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:40.530254 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" podUID="90bfb1e4-9526-40f6-aed4-ee46c2d86471" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Oct 30 21:46:40.643235 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:40.643129 4560 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" podUID="e3281dbe-e8f9-4bab-9ae4-2279e79b7292" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:40.660371 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:40.651600 4560 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" podUID="0155a02e-8df7-4847-b40f-b8d9e4bac8c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:40.805275 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:40.804429 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" podUID="b80b0fba-3830-4a74-a724-aead35443b89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:40.826498 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:40.805571 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" Oct 30 21:46:40.845373 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:40.843820 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" podUID="4dee11ef-9647-47ee-a4b1-686680256542" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:40.845373 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:40.844599 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" podUID="e3281dbe-e8f9-4bab-9ae4-2279e79b7292" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:40.849360 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:40.847036 4560 patch_prober.go:29] interesting pod/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 30 21:46:40.849360 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:40.847087 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" podUID="1bfd4837-aa59-491c-8e32-770bf1451706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 30 21:46:40.862362 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:40.859408 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" podUID="1e044c02-0053-40e8-9f80-e806cd2d9ae5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:41.034489 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:40.989659 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" podUID="0155a02e-8df7-4847-b40f-b8d9e4bac8c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded" Oct 30 21:46:41.034489 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:40.990465 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" Oct 30 21:46:41.107371 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:41.012857 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" podUID="971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:41.107371 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:41.091318 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" podUID="9e2d7535-075d-4211-b814-35d6abff7605" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:41.107371 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:41.091742 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" podUID="0ff8ae26-4364-4c4e-9285-721d1a486a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 21:46:41.107371 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:41.092110 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" Oct 30 21:46:41.107371 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:41.092194 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" Oct 30 21:46:41.107371 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:41.095144 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" podUID="27df5dfe-47a4-4ebb-bd09-1ceef406600a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:41.107371 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:41.095443 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" Oct 30 21:46:41.107371 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:41.096269 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" podUID="a59a0fc3-8316-47eb-a42e-552d202b1ab7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 30 21:46:41.142459 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:41.112693 4560 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" podUID="971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": context deadline exceeded" Oct 30 21:46:41.257047 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:41.159803 4560 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:46:41.257047 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:41.159887 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" podUID="a0802666-5926-4a4f-a438-8e932494758c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:41.257047 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:41.168495 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" Oct 30 21:46:41.257047 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:41.197012 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" podUID="45a5bac0-28ab-4c78-856e-303f6d4522de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": context deadline exceeded" Oct 30 21:46:41.257047 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:41.206726 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" Oct 30 21:46:41.295863 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:41.295770 4560 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:46:41.295863 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:41.295836 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" podUID="5d83608f-bcb0-4aab-b14e-29846561a3d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:41.383829 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:41.335686 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" podUID="8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:41.383829 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:41.336142 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" Oct 30 21:46:41.383829 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:41.345375 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" Oct 30 21:46:41.517492 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:41.517444 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" podUID="e974bae1-bb18-46b0-a4ee-4c01d6455e32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:41.533590 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:41.533564 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" Oct 30 21:46:41.537976 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:41.537943 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" podUID="90bfb1e4-9526-40f6-aed4-ee46c2d86471" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:41.538759 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:41.538741 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" Oct 30 21:46:41.718503 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:41.718465 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1f2444-7ss99" event={"ID":"a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806","Type":"ContainerStarted","Data":"b67046ac1eb6c4e24d2fce0ebc150982876813994c5d4e46a9d077921653d959"} Oct 30 21:46:41.941023 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:41.917027 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" podUID="b80b0fba-3830-4a74-a724-aead35443b89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:41.941023 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:41.939576 4560 patch_prober.go:29] interesting pod/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 30 21:46:41.941023 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:41.939635 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" podUID="1bfd4837-aa59-491c-8e32-770bf1451706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 30 21:46:41.965368 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:41.955953 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" podUID="e3281dbe-e8f9-4bab-9ae4-2279e79b7292" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:41.965368 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:41.956309 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" podUID="4dee11ef-9647-47ee-a4b1-686680256542" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:41.965368 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:41.956620 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" Oct 30 21:46:41.965368 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:41.956681 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" Oct 30 21:46:41.994825 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:41.994437 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" podUID="0155a02e-8df7-4847-b40f-b8d9e4bac8c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded" Oct 30 21:46:42.077199 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:42.077153 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" podUID="971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:42.130366 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:42.105829 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" podUID="a59a0fc3-8316-47eb-a42e-552d202b1ab7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 30 21:46:42.155365 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:42.107867 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" Oct 30 21:46:42.180364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:42.108684 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" podUID="0ff8ae26-4364-4c4e-9285-721d1a486a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:42.189363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:42.108717 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" podUID="9e2d7535-075d-4211-b814-35d6abff7605" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:46:42.197416 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:42.116746 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78ca65b-hbdf2" Oct 30 21:46:42.205468 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:42.192462 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" Oct 30 21:46:42.209496 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:42.209472 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" Oct 30 21:46:42.234430 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:42.208802 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" podUID="45a5bac0-28ab-4c78-856e-303f6d4522de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:42.346363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:42.333511 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd250b4b-880a-47b1-bd3c-7843bc9d0f28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd250b4b-880a-47b1-bd3c-7843bc9d0f28-maas-api-xks-ca\") pod \"fd250b4b-880a-47b1-bd3c-7843bc9d0f28\" (UID: \"fd250b4b-880a-47b1-bd3c-7843bc9d0f28\") " Oct 30 21:46:42.346363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:42.333592 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd250b4b-880a-47b1-bd3c-7843bc9d0f28-kube-api-access-gx7wp\" (UniqueName: \"kubernetes.io/projected/fd250b4b-880a-47b1-bd3c-7843bc9d0f28-kube-api-access-gx7wp\") pod \"fd250b4b-880a-47b1-bd3c-7843bc9d0f28\" (UID: \"fd250b4b-880a-47b1-bd3c-7843bc9d0f28\") " Oct 30 21:46:42.346363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:42.333679 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd250b4b-880a-47b1-bd3c-7843bc9d0f28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd250b4b-880a-47b1-bd3c-7843bc9d0f28-maas-api-openshift-ca\") pod \"fd250b4b-880a-47b1-bd3c-7843bc9d0f28\" (UID: \"fd250b4b-880a-47b1-bd3c-7843bc9d0f28\") " Oct 30 21:46:42.377365 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:42.376670 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd250b4b-880a-47b1-bd3c-7843bc9d0f28-maas-api-openshift-ca" pod "fd250b4b-880a-47b1-bd3c-7843bc9d0f28" (UID: "fd250b4b-880a-47b1-bd3c-7843bc9d0f28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:42.383369 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:42.379321 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" Oct 30 21:46:42.470416 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:42.466788 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd250b4b-880a-47b1-bd3c-7843bc9d0f28-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:42.541234 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:42.540850 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" podUID="90bfb1e4-9526-40f6-aed4-ee46c2d86471" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:42.630429 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:42.628527 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd250b4b-880a-47b1-bd3c-7843bc9d0f28-kube-api-access-gx7wp" pod "fd250b4b-880a-47b1-bd3c-7843bc9d0f28" (UID: "fd250b4b-880a-47b1-bd3c-7843bc9d0f28"). InnerVolumeSpecName "kube-api-access-gx7wp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:42.631090 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:42.631039 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx7wp\" (UniqueName: \"kubernetes.io/projected/fd250b4b-880a-47b1-bd3c-7843bc9d0f28-kube-api-access-gx7wp\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:42.634213 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:42.634169 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd250b4b-880a-47b1-bd3c-7843bc9d0f28-maas-api-xks-ca" pod "fd250b4b-880a-47b1-bd3c-7843bc9d0f28" (UID: "fd250b4b-880a-47b1-bd3c-7843bc9d0f28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:42.727597 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:42.725990 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" Oct 30 21:46:42.727597 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:42.726119 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" Oct 30 21:46:42.784375 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:42.782038 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd250b4b-880a-47b1-bd3c-7843bc9d0f28-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:42.895052 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:42.894977 4560 patch_prober.go:29] interesting pod/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 30 21:46:42.902197 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:42.902127 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" podUID="1bfd4837-aa59-491c-8e32-770bf1451706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 30 21:46:42.957041 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:42.957000 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" podUID="4dee11ef-9647-47ee-a4b1-686680256542" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:42.957939 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:42.957588 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" podUID="e3281dbe-e8f9-4bab-9ae4-2279e79b7292" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 30 21:46:42.957939 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:42.957877 4560 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" podUID="45a5bac0-28ab-4c78-856e-303f6d4522de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:43.049807 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:43.049709 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" podUID="0155a02e-8df7-4847-b40f-b8d9e4bac8c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:43.237398 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:43.228485 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" podUID="a59a0fc3-8316-47eb-a42e-552d202b1ab7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:43.262903 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:43.052727 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1f2444-7ss99" event={"ID":"a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806","Type":"ContainerDied","Data":"13a15df29b897c7e583c5c975d5d009fb980c5a2671964587df825a2bae247f7"} Oct 30 21:46:43.262903 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:43.260741 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" podUID="8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:43.262903 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:43.052697 4560 generic.go:356] "Generic (PLEG): container finished" podID="a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806" containerID="13a15df29b897c7e583c5c975d5d009fb980c5a2671964587df825a2bae247f7" exitCode=0 Oct 30 21:46:43.289408 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:43.285018 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" podUID="971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 21:46:43.289408 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:43.286006 4560 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" podUID="1e044c02-0053-40e8-9f80-e806cd2d9ae5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:43.296455 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:43.296418 4560 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:46:43.296615 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:43.296477 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" podUID="d64f50e0-cf7a-43ec-83ac-a73c0972cb39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:43.342524 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:43.342431 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" podUID="0ff8ae26-4364-4c4e-9285-721d1a486a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:43.343553 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:43.343524 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78ca65b-hbdf2" event={"ID":"fd250b4b-880a-47b1-bd3c-7843bc9d0f28","Type":"ContainerDied","Data":"39cf9086bc38e8834e6a7d76eefa5a749ff96b5e733ccd57f9c620d9f68ad1c7"} Oct 30 21:46:43.343749 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:43.343732 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39cf9086bc38e8834e6a7d76eefa5a749ff96b5e733ccd57f9c620d9f68ad1c7" Oct 30 21:46:43.343923 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:43.343909 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78ca65b-hbdf2" Oct 30 21:46:43.461115 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:43.461057 4560 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" podUID="e974bae1-bb18-46b0-a4ee-4c01d6455e32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:43.524815 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:43.524770 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" podUID="45a5bac0-28ab-4c78-856e-303f6d4522de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:43.526957 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:43.526922 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" podUID="27df5dfe-47a4-4ebb-bd09-1ceef406600a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:43.568374 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:43.565715 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" podUID="90bfb1e4-9526-40f6-aed4-ee46c2d86471" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:43.688613 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:43.687991 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" podUID="9e2d7535-075d-4211-b814-35d6abff7605" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:43.774908 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:43.774432 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" podUID="1e044c02-0053-40e8-9f80-e806cd2d9ae5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:43.851403 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:43.849597 4560 patch_prober.go:29] interesting pod/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 30 21:46:43.851403 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:43.849658 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" podUID="1bfd4837-aa59-491c-8e32-770bf1451706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 30 21:46:43.879363 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:46:43.878973 4560 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9dc400f_0aa4_4ccf_a93d_81f3c8bf4806.slice/crio-conmon-13a15df29b897c7e583c5c975d5d009fb980c5a2671964587df825a2bae247f7.scope\": RecentStats: unable to find data in memory cache]" Oct 30 21:46:44.021974 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:43.970878 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" Oct 30 21:46:44.021974 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:43.972076 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" podUID="e3281dbe-e8f9-4bab-9ae4-2279e79b7292" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:44.021974 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:43.989713 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" podUID="4dee11ef-9647-47ee-a4b1-686680256542" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Oct 30 21:46:44.021974 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:44.002486 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" podUID="b80b0fba-3830-4a74-a724-aead35443b89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:44.022289 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:44.022031 4560 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" podUID="b80b0fba-3830-4a74-a724-aead35443b89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:44.226993 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:44.225492 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" podUID="e974bae1-bb18-46b0-a4ee-4c01d6455e32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 21:46:44.226993 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:44.226127 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" Oct 30 21:46:44.267287 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:44.266118 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" podUID="a59a0fc3-8316-47eb-a42e-552d202b1ab7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:44.287270 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:44.286576 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" podUID="971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:44.289117 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:44.288260 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" Oct 30 21:46:44.302368 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:44.296387 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" Oct 30 21:46:44.302595 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:44.302569 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" podUID="8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:44.395729 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:44.395696 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" Oct 30 21:46:44.473260 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:44.473209 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" Oct 30 21:46:44.473581 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:44.473563 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" Oct 30 21:46:44.526383 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:44.525315 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" podUID="45a5bac0-28ab-4c78-856e-303f6d4522de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:44.527232 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:44.527203 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" Oct 30 21:46:44.527478 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:44.527445 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" podUID="27df5dfe-47a4-4ebb-bd09-1ceef406600a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:44.527841 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:44.527821 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" Oct 30 21:46:44.634754 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:44.626548 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" Oct 30 21:46:44.634754 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:44.626630 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" Oct 30 21:46:44.634754 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:44.626661 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" Oct 30 21:46:44.634754 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:44.626689 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" Oct 30 21:46:44.795019 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:44.794995 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1f2444-7ss99" Oct 30 21:46:44.839262 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:44.839235 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806-maas-api-xks-ca\") pod \"a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806\" (UID: \"a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806\") " Oct 30 21:46:44.839467 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:44.839281 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806-maas-api-openshift-ca\") pod \"a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806\" (UID: \"a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806\") " Oct 30 21:46:44.839467 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:44.839366 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806-kube-api-access-974h2\" (UniqueName: \"kubernetes.io/projected/a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806-kube-api-access-974h2\") pod \"a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806\" (UID: \"a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806\") " Oct 30 21:46:44.839716 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:44.839692 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806-maas-api-openshift-ca" pod "a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806" (UID: "a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:44.839838 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:44.839823 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:44.841532 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:44.841510 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806-maas-api-xks-ca" pod "a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806" (UID: "a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:44.841813 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:44.841785 4560 patch_prober.go:29] interesting pod/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 30 21:46:44.841890 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:44.841841 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" podUID="1bfd4837-aa59-491c-8e32-770bf1451706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 30 21:46:44.841890 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:44.841877 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806-kube-api-access-974h2" pod "a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806" (UID: "a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806"). InnerVolumeSpecName "kube-api-access-974h2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:44.941677 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:44.941605 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:44.941677 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:44.941632 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-974h2\" (UniqueName: \"kubernetes.io/projected/a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806-kube-api-access-974h2\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:45.068861 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:45.068832 4560 scope.go:122] "RemoveContainer" containerID="6832ec27c668e040bba774ffad4fc0d5dc0d67a9822e242ae356b4a109d994fe" Oct 30 21:46:45.082784 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:45.082754 4560 scope.go:122] "RemoveContainer" containerID="752ae82acd8176b28cb4ad563c799fc7f198866aae51cd3aee70028895bc4c41" Oct 30 21:46:45.093999 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:45.093975 4560 scope.go:122] "RemoveContainer" containerID="e0c4e8b8457c9b5a77b79bd2e5b49fafae11ec178d608c33346c84e4a3e2ce45" Oct 30 21:46:45.104765 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:45.104738 4560 scope.go:122] "RemoveContainer" containerID="77d05270c42c708d17bb96b3f8eae241fa655ef1103be12c586f0034b8183460" Oct 30 21:46:45.114748 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:45.114727 4560 scope.go:122] "RemoveContainer" containerID="94565846875b3db65cf8357d9a823cf23a4f6a4a2abbaa94d8b10c30a3651f17" Oct 30 21:46:45.124076 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:45.124057 4560 scope.go:122] "RemoveContainer" containerID="c257fc7912bb943b0e68d5a49d31eac41d59c429e2f2560c2b51ded88dd7adfd" Oct 30 21:46:45.135886 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:45.135865 4560 scope.go:122] "RemoveContainer" containerID="6146e0f05a9d978b43290b1d6cb83093286d8b71fd4a2bd1b33641d35ec3eee3" Oct 30 21:46:45.151011 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:45.150980 4560 scope.go:122] "RemoveContainer" containerID="3ad254a96d9b853c2d3c74699e26b21d96712a20b0281bc572f4fba5e83d0515" Oct 30 21:46:45.160735 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:45.160710 4560 scope.go:122] "RemoveContainer" containerID="ed8a7b55cb438a06a14a71b785840faeb57ed319dc362d3db65b521a621be380" Oct 30 21:46:45.169351 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:45.169317 4560 scope.go:122] "RemoveContainer" containerID="d28b7a41851f31dc1d3f160f1027bffe50ed095b1b670c8eab86d4caa917e0ce" Oct 30 21:46:45.177786 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:45.177770 4560 scope.go:122] "RemoveContainer" containerID="ceb54cb8b75ed349d77c81843fa396176328a7e98f34406e1b7911849f787fa8" Oct 30 21:46:45.287259 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:45.286163 4560 scope.go:122] "RemoveContainer" containerID="e53f80448d86d45eeb68d7aded90a60fdb1edc535a58bb6fd089ec81abf4568b" Oct 30 21:46:45.491527 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:45.491497 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1f2444-7ss99" event={"ID":"a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806","Type":"ContainerDied","Data":"b67046ac1eb6c4e24d2fce0ebc150982876813994c5d4e46a9d077921653d959"} Oct 30 21:46:45.491631 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:45.491557 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b67046ac1eb6c4e24d2fce0ebc150982876813994c5d4e46a9d077921653d959" Oct 30 21:46:45.491631 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:45.491504 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1f2444-7ss99" Oct 30 21:46:46.366902 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:46.366861 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" Oct 30 21:46:46.368099 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:46.367962 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" Oct 30 21:46:46.374303 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:46.374263 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" Oct 30 21:46:46.598009 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:46.597979 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62bafda8-wtgtq"] Oct 30 21:46:46.598465 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:46.598452 4560 cpu_manager.go:414] "containerMap: removing container" podUID="fd250b4b-880a-47b1-bd3c-7843bc9d0f28" containerName="revoke-keys" Oct 30 21:46:46.598507 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:46.598467 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd250b4b-880a-47b1-bd3c-7843bc9d0f28" containerName="revoke-keys" Oct 30 21:46:46.598555 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:46.598507 4560 cpu_manager.go:414] "containerMap: removing container" podUID="a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806" containerName="revoke-keys" Oct 30 21:46:46.598555 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:46.598514 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806" containerName="revoke-keys" Oct 30 21:46:46.598611 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:46.598589 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806" containerName="revoke-keys" Oct 30 21:46:46.598611 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:46.598599 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd250b4b-880a-47b1-bd3c-7843bc9d0f28" containerName="revoke-keys" Oct 30 21:46:46.604625 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:46.604575 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62bafda8-wtgtq" Oct 30 21:46:46.655180 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:46.655090 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:46:46.655180 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:46.655129 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:46:46.655531 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:46.655510 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:46:46.655704 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:46.655675 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:46:46.657376 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:46.657354 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62bafda8-wtgtq"] Oct 30 21:46:46.695693 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:46.695670 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62bafda8-wtgtq\" (UID: \"5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62bafda8-wtgtq" Oct 30 21:46:46.695822 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:46.695738 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdv62\" (UniqueName: \"kubernetes.io/projected/5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39-kube-api-access-cdv62\") pod \"maas-api-revoke-sub-62bafda8-wtgtq\" (UID: \"5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62bafda8-wtgtq" Oct 30 21:46:46.695896 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:46.695878 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62bafda8-wtgtq\" (UID: \"5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62bafda8-wtgtq" Oct 30 21:46:46.797297 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:46.797266 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62bafda8-wtgtq\" (UID: \"5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62bafda8-wtgtq" Oct 30 21:46:46.797486 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:46.797388 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdv62\" (UniqueName: \"kubernetes.io/projected/5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39-kube-api-access-cdv62\") pod \"maas-api-revoke-sub-62bafda8-wtgtq\" (UID: \"5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62bafda8-wtgtq" Oct 30 21:46:46.797486 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:46.797468 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62bafda8-wtgtq\" (UID: \"5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62bafda8-wtgtq" Oct 30 21:46:46.798400 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:46.798371 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62bafda8-wtgtq\" (UID: \"5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62bafda8-wtgtq" Oct 30 21:46:46.799818 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:46.799796 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62bafda8-wtgtq\" (UID: \"5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62bafda8-wtgtq" Oct 30 21:46:46.857124 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:46.857098 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdv62\" (UniqueName: \"kubernetes.io/projected/5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39-kube-api-access-cdv62\") pod \"maas-api-revoke-sub-62bafda8-wtgtq\" (UID: \"5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62bafda8-wtgtq" Oct 30 21:46:46.925683 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:46.925538 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62bafda8-wtgtq" Oct 30 21:46:47.477288 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:47.477243 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62bafda8-wtgtq"] Oct 30 21:46:47.481268 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:46:47.481239 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a6f8fc4_1854_4ef1_aa67_0cb4c1627e39.slice/crio-a7cf058f9bc2b54fc25799239d999e6e8dea729553e343bcfb686a5e5cb24746 WatchSource:0}: Error finding container a7cf058f9bc2b54fc25799239d999e6e8dea729553e343bcfb686a5e5cb24746: Status 404 returned error can't find the container with id a7cf058f9bc2b54fc25799239d999e6e8dea729553e343bcfb686a5e5cb24746 Oct 30 21:46:47.529229 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:47.529194 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62bafda8-wtgtq" event={"ID":"5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39","Type":"ContainerStarted","Data":"a7cf058f9bc2b54fc25799239d999e6e8dea729553e343bcfb686a5e5cb24746"} Oct 30 21:46:48.538689 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:48.538649 4560 generic.go:356] "Generic (PLEG): container finished" podID="5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39" containerID="e3f3ccd8183b96ed390b8205ed335dbdf53d896cb087e6b923b809efa46b4855" exitCode=0 Oct 30 21:46:48.539071 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:48.538708 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62bafda8-wtgtq" event={"ID":"5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39","Type":"ContainerDied","Data":"e3f3ccd8183b96ed390b8205ed335dbdf53d896cb087e6b923b809efa46b4855"} Oct 30 21:46:48.875141 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:48.875107 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9217b2b5-hxxgv"] Oct 30 21:46:48.879695 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:48.879678 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9217b2b5-hxxgv" Oct 30 21:46:48.933593 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:48.933568 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9217b2b5-hxxgv"] Oct 30 21:46:48.942320 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:48.942293 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d25781de-5282-4af7-8199-a1fa695f19ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9217b2b5-hxxgv\" (UID: \"d25781de-5282-4af7-8199-a1fa695f19ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9217b2b5-hxxgv" Oct 30 21:46:48.942456 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:48.942376 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d25781de-5282-4af7-8199-a1fa695f19ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9217b2b5-hxxgv\" (UID: \"d25781de-5282-4af7-8199-a1fa695f19ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9217b2b5-hxxgv" Oct 30 21:46:48.942564 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:48.942545 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsdtc\" (UniqueName: \"kubernetes.io/projected/d25781de-5282-4af7-8199-a1fa695f19ba-kube-api-access-wsdtc\") pod \"maas-api-revoke-sub-9217b2b5-hxxgv\" (UID: \"d25781de-5282-4af7-8199-a1fa695f19ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9217b2b5-hxxgv" Oct 30 21:46:49.048363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:49.043678 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsdtc\" (UniqueName: \"kubernetes.io/projected/d25781de-5282-4af7-8199-a1fa695f19ba-kube-api-access-wsdtc\") pod \"maas-api-revoke-sub-9217b2b5-hxxgv\" (UID: \"d25781de-5282-4af7-8199-a1fa695f19ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9217b2b5-hxxgv" Oct 30 21:46:49.054417 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:49.051413 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d25781de-5282-4af7-8199-a1fa695f19ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9217b2b5-hxxgv\" (UID: \"d25781de-5282-4af7-8199-a1fa695f19ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9217b2b5-hxxgv" Oct 30 21:46:49.054417 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:49.051557 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d25781de-5282-4af7-8199-a1fa695f19ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9217b2b5-hxxgv\" (UID: \"d25781de-5282-4af7-8199-a1fa695f19ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9217b2b5-hxxgv" Oct 30 21:46:49.054417 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:49.052714 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d25781de-5282-4af7-8199-a1fa695f19ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9217b2b5-hxxgv\" (UID: \"d25781de-5282-4af7-8199-a1fa695f19ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9217b2b5-hxxgv" Oct 30 21:46:49.087363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:49.075953 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d25781de-5282-4af7-8199-a1fa695f19ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9217b2b5-hxxgv\" (UID: \"d25781de-5282-4af7-8199-a1fa695f19ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9217b2b5-hxxgv" Oct 30 21:46:49.121105 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:49.118823 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsdtc\" (UniqueName: \"kubernetes.io/projected/d25781de-5282-4af7-8199-a1fa695f19ba-kube-api-access-wsdtc\") pod \"maas-api-revoke-sub-9217b2b5-hxxgv\" (UID: \"d25781de-5282-4af7-8199-a1fa695f19ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9217b2b5-hxxgv" Oct 30 21:46:49.192879 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:49.192801 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9217b2b5-hxxgv" Oct 30 21:46:49.598151 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:49.598118 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9217b2b5-hxxgv"] Oct 30 21:46:49.601874 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:46:49.601833 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd25781de_5282_4af7_8199_a1fa695f19ba.slice/crio-3b4935399c4a51b0567d983a99d9a2905466d4775a295e5e7ad6c77167dfe7bf WatchSource:0}: Error finding container 3b4935399c4a51b0567d983a99d9a2905466d4775a295e5e7ad6c77167dfe7bf: Status 404 returned error can't find the container with id 3b4935399c4a51b0567d983a99d9a2905466d4775a295e5e7ad6c77167dfe7bf Oct 30 21:46:49.686532 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:46:49.686500 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:46:49.710686 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:49.710665 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62bafda8-wtgtq" Oct 30 21:46:49.759582 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:49.759553 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39-maas-api-openshift-ca\") pod \"5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39\" (UID: \"5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39\") " Oct 30 21:46:49.759727 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:49.759593 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39-kube-api-access-cdv62\" (UniqueName: \"kubernetes.io/projected/5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39-kube-api-access-cdv62\") pod \"5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39\" (UID: \"5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39\") " Oct 30 21:46:49.759794 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:49.759754 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39-maas-api-xks-ca\") pod \"5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39\" (UID: \"5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39\") " Oct 30 21:46:49.759934 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:49.759912 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39-maas-api-openshift-ca" pod "5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39" (UID: "5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:49.760172 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:49.760148 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:49.761697 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:49.761675 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39-maas-api-xks-ca" pod "5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39" (UID: "5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:49.761697 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:49.761690 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39-kube-api-access-cdv62" pod "5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39" (UID: "5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39"). InnerVolumeSpecName "kube-api-access-cdv62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:49.861562 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:49.861530 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:49.861689 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:49.861565 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdv62\" (UniqueName: \"kubernetes.io/projected/5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39-kube-api-access-cdv62\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:50.560908 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:50.560873 4560 generic.go:356] "Generic (PLEG): container finished" podID="d25781de-5282-4af7-8199-a1fa695f19ba" containerID="47fbdc66fda9184e7943037954c278bbea0fbd605323402ffd363851d46d5893" exitCode=0 Oct 30 21:46:50.561010 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:50.560954 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9217b2b5-hxxgv" event={"ID":"d25781de-5282-4af7-8199-a1fa695f19ba","Type":"ContainerDied","Data":"47fbdc66fda9184e7943037954c278bbea0fbd605323402ffd363851d46d5893"} Oct 30 21:46:50.561054 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:50.561040 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9217b2b5-hxxgv" event={"ID":"d25781de-5282-4af7-8199-a1fa695f19ba","Type":"ContainerStarted","Data":"3b4935399c4a51b0567d983a99d9a2905466d4775a295e5e7ad6c77167dfe7bf"} Oct 30 21:46:50.562404 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:50.562386 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62bafda8-wtgtq" Oct 30 21:46:50.562501 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:50.562382 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62bafda8-wtgtq" event={"ID":"5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39","Type":"ContainerDied","Data":"a7cf058f9bc2b54fc25799239d999e6e8dea729553e343bcfb686a5e5cb24746"} Oct 30 21:46:50.562564 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:50.562507 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7cf058f9bc2b54fc25799239d999e6e8dea729553e343bcfb686a5e5cb24746" Oct 30 21:46:51.727125 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:51.727106 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9217b2b5-hxxgv" Oct 30 21:46:51.781390 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:51.781366 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d25781de-5282-4af7-8199-a1fa695f19ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d25781de-5282-4af7-8199-a1fa695f19ba-maas-api-openshift-ca\") pod \"d25781de-5282-4af7-8199-a1fa695f19ba\" (UID: \"d25781de-5282-4af7-8199-a1fa695f19ba\") " Oct 30 21:46:51.781506 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:51.781489 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d25781de-5282-4af7-8199-a1fa695f19ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d25781de-5282-4af7-8199-a1fa695f19ba-maas-api-xks-ca\") pod \"d25781de-5282-4af7-8199-a1fa695f19ba\" (UID: \"d25781de-5282-4af7-8199-a1fa695f19ba\") " Oct 30 21:46:51.781566 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:51.781515 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d25781de-5282-4af7-8199-a1fa695f19ba-kube-api-access-wsdtc\" (UniqueName: \"kubernetes.io/projected/d25781de-5282-4af7-8199-a1fa695f19ba-kube-api-access-wsdtc\") pod \"d25781de-5282-4af7-8199-a1fa695f19ba\" (UID: \"d25781de-5282-4af7-8199-a1fa695f19ba\") " Oct 30 21:46:51.781732 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:51.781708 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d25781de-5282-4af7-8199-a1fa695f19ba-maas-api-openshift-ca" pod "d25781de-5282-4af7-8199-a1fa695f19ba" (UID: "d25781de-5282-4af7-8199-a1fa695f19ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:51.781965 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:51.781949 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d25781de-5282-4af7-8199-a1fa695f19ba-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:51.783457 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:51.783435 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d25781de-5282-4af7-8199-a1fa695f19ba-maas-api-xks-ca" pod "d25781de-5282-4af7-8199-a1fa695f19ba" (UID: "d25781de-5282-4af7-8199-a1fa695f19ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:51.783457 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:51.783448 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d25781de-5282-4af7-8199-a1fa695f19ba-kube-api-access-wsdtc" pod "d25781de-5282-4af7-8199-a1fa695f19ba" (UID: "d25781de-5282-4af7-8199-a1fa695f19ba"). InnerVolumeSpecName "kube-api-access-wsdtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:51.883028 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:51.882955 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d25781de-5282-4af7-8199-a1fa695f19ba-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:51.883028 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:51.882993 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsdtc\" (UniqueName: \"kubernetes.io/projected/d25781de-5282-4af7-8199-a1fa695f19ba-kube-api-access-wsdtc\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:52.580935 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:52.580900 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9217b2b5-hxxgv" event={"ID":"d25781de-5282-4af7-8199-a1fa695f19ba","Type":"ContainerDied","Data":"3b4935399c4a51b0567d983a99d9a2905466d4775a295e5e7ad6c77167dfe7bf"} Oct 30 21:46:52.580935 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:52.580927 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9217b2b5-hxxgv" Oct 30 21:46:52.581167 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:46:52.580950 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b4935399c4a51b0567d983a99d9a2905466d4775a295e5e7ad6c77167dfe7bf" Oct 30 21:47:00.897313 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:00.897277 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq"] Oct 30 21:47:00.897853 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:00.897836 4560 cpu_manager.go:414] "containerMap: removing container" podUID="d25781de-5282-4af7-8199-a1fa695f19ba" containerName="revoke-keys" Oct 30 21:47:00.897896 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:00.897854 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d25781de-5282-4af7-8199-a1fa695f19ba" containerName="revoke-keys" Oct 30 21:47:00.897896 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:00.897881 4560 cpu_manager.go:414] "containerMap: removing container" podUID="5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39" containerName="revoke-keys" Oct 30 21:47:00.897896 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:00.897886 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39" containerName="revoke-keys" Oct 30 21:47:00.897992 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:00.897972 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39" containerName="revoke-keys" Oct 30 21:47:00.897992 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:00.897981 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="d25781de-5282-4af7-8199-a1fa695f19ba" containerName="revoke-keys" Oct 30 21:47:00.901863 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:00.901846 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" Oct 30 21:47:00.972892 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:00.972856 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-364e1518-openshift-default-dockercfg-6zn9t\"" Oct 30 21:47:01.156452 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:01.103297 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b247p\" (UniqueName: \"kubernetes.io/projected/5c0221dd-a19a-464f-93af-996a4a53a62a-kube-api-access-b247p\") pod \"e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq\" (UID: \"5c0221dd-a19a-464f-93af-996a4a53a62a\") " pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" Oct 30 21:47:01.156452 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:01.103447 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c0221dd-a19a-464f-93af-996a4a53a62a-istiod-ca-cert\") pod \"e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq\" (UID: \"5c0221dd-a19a-464f-93af-996a4a53a62a\") " pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" Oct 30 21:47:01.156452 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:01.103565 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0221dd-a19a-464f-93af-996a4a53a62a-workload-socket\") pod \"e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq\" (UID: \"5c0221dd-a19a-464f-93af-996a4a53a62a\") " pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" Oct 30 21:47:01.156452 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:01.103614 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c0221dd-a19a-464f-93af-996a4a53a62a-istio-data\") pod \"e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq\" (UID: \"5c0221dd-a19a-464f-93af-996a4a53a62a\") " pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" Oct 30 21:47:01.156452 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:01.103673 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c0221dd-a19a-464f-93af-996a4a53a62a-istio-token\") pod \"e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq\" (UID: \"5c0221dd-a19a-464f-93af-996a4a53a62a\") " pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" Oct 30 21:47:01.156452 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:01.103713 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c0221dd-a19a-464f-93af-996a4a53a62a-istio-envoy\") pod \"e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq\" (UID: \"5c0221dd-a19a-464f-93af-996a4a53a62a\") " pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" Oct 30 21:47:01.156452 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:01.103737 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c0221dd-a19a-464f-93af-996a4a53a62a-istio-podinfo\") pod \"e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq\" (UID: \"5c0221dd-a19a-464f-93af-996a4a53a62a\") " pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" Oct 30 21:47:01.156452 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:01.103775 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0221dd-a19a-464f-93af-996a4a53a62a-credential-socket\") pod \"e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq\" (UID: \"5c0221dd-a19a-464f-93af-996a4a53a62a\") " pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" Oct 30 21:47:01.156452 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:01.103804 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c0221dd-a19a-464f-93af-996a4a53a62a-workload-certs\") pod \"e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq\" (UID: \"5c0221dd-a19a-464f-93af-996a4a53a62a\") " pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" Oct 30 21:47:01.163349 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:01.163304 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq"] Oct 30 21:47:01.218363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:01.214290 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0221dd-a19a-464f-93af-996a4a53a62a-workload-socket\") pod \"e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq\" (UID: \"5c0221dd-a19a-464f-93af-996a4a53a62a\") " pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" Oct 30 21:47:01.233357 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:01.221877 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0221dd-a19a-464f-93af-996a4a53a62a-workload-socket\") pod \"e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq\" (UID: \"5c0221dd-a19a-464f-93af-996a4a53a62a\") " pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" Oct 30 21:47:01.257364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:01.245583 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c0221dd-a19a-464f-93af-996a4a53a62a-istio-data\") pod \"e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq\" (UID: \"5c0221dd-a19a-464f-93af-996a4a53a62a\") " pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" Oct 30 21:47:01.257364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:01.245678 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c0221dd-a19a-464f-93af-996a4a53a62a-istio-token\") pod \"e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq\" (UID: \"5c0221dd-a19a-464f-93af-996a4a53a62a\") " pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" Oct 30 21:47:01.257364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:01.245721 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c0221dd-a19a-464f-93af-996a4a53a62a-istio-envoy\") pod \"e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq\" (UID: \"5c0221dd-a19a-464f-93af-996a4a53a62a\") " pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" Oct 30 21:47:01.257364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:01.245742 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c0221dd-a19a-464f-93af-996a4a53a62a-istio-podinfo\") pod \"e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq\" (UID: \"5c0221dd-a19a-464f-93af-996a4a53a62a\") " pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" Oct 30 21:47:01.257364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:01.245779 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0221dd-a19a-464f-93af-996a4a53a62a-credential-socket\") pod \"e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq\" (UID: \"5c0221dd-a19a-464f-93af-996a4a53a62a\") " pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" Oct 30 21:47:01.257364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:01.245805 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c0221dd-a19a-464f-93af-996a4a53a62a-workload-certs\") pod \"e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq\" (UID: \"5c0221dd-a19a-464f-93af-996a4a53a62a\") " pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" Oct 30 21:47:01.257364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:01.245909 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b247p\" (UniqueName: \"kubernetes.io/projected/5c0221dd-a19a-464f-93af-996a4a53a62a-kube-api-access-b247p\") pod \"e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq\" (UID: \"5c0221dd-a19a-464f-93af-996a4a53a62a\") " pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" Oct 30 21:47:01.257364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:01.246047 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c0221dd-a19a-464f-93af-996a4a53a62a-istiod-ca-cert\") pod \"e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq\" (UID: \"5c0221dd-a19a-464f-93af-996a4a53a62a\") " pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" Oct 30 21:47:01.257364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:01.246957 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c0221dd-a19a-464f-93af-996a4a53a62a-istiod-ca-cert\") pod \"e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq\" (UID: \"5c0221dd-a19a-464f-93af-996a4a53a62a\") " pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" Oct 30 21:47:01.257364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:01.247243 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c0221dd-a19a-464f-93af-996a4a53a62a-istio-data\") pod \"e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq\" (UID: \"5c0221dd-a19a-464f-93af-996a4a53a62a\") " pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" Oct 30 21:47:01.259193 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:01.259163 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c0221dd-a19a-464f-93af-996a4a53a62a-workload-certs\") pod \"e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq\" (UID: \"5c0221dd-a19a-464f-93af-996a4a53a62a\") " pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" Oct 30 21:47:01.290766 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:01.290728 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0221dd-a19a-464f-93af-996a4a53a62a-credential-socket\") pod \"e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq\" (UID: \"5c0221dd-a19a-464f-93af-996a4a53a62a\") " pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" Oct 30 21:47:01.307365 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:01.302445 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c0221dd-a19a-464f-93af-996a4a53a62a-istio-envoy\") pod \"e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq\" (UID: \"5c0221dd-a19a-464f-93af-996a4a53a62a\") " pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" Oct 30 21:47:01.313368 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:01.312552 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c0221dd-a19a-464f-93af-996a4a53a62a-istio-podinfo\") pod \"e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq\" (UID: \"5c0221dd-a19a-464f-93af-996a4a53a62a\") " pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" Oct 30 21:47:01.333612 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:01.333533 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b247p\" (UniqueName: \"kubernetes.io/projected/5c0221dd-a19a-464f-93af-996a4a53a62a-kube-api-access-b247p\") pod \"e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq\" (UID: \"5c0221dd-a19a-464f-93af-996a4a53a62a\") " pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" Oct 30 21:47:01.364207 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:01.362939 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c0221dd-a19a-464f-93af-996a4a53a62a-istio-token\") pod \"e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq\" (UID: \"5c0221dd-a19a-464f-93af-996a4a53a62a\") " pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" Oct 30 21:47:01.518874 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:01.517592 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" Oct 30 21:47:02.026869 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:02.026836 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq"] Oct 30 21:47:02.031424 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:47:02.031399 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c0221dd_a19a_464f_93af_996a4a53a62a.slice/crio-173b822715915167cb38f5898a576822d50a8432396ff4e7178e73d705541818 WatchSource:0}: Error finding container 173b822715915167cb38f5898a576822d50a8432396ff4e7178e73d705541818: Status 404 returned error can't find the container with id 173b822715915167cb38f5898a576822d50a8432396ff4e7178e73d705541818 Oct 30 21:47:02.034492 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:02.034455 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:47:02.034601 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:02.034541 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:47:02.034676 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:02.034600 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:47:02.695325 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:02.695281 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" event={"ID":"5c0221dd-a19a-464f-93af-996a4a53a62a","Type":"ContainerStarted","Data":"3b3013fa31a125cb8facbd94d31e33324f30129daa8160f1fa385b10c36fbc6e"} Oct 30 21:47:02.695544 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:02.695365 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" event={"ID":"5c0221dd-a19a-464f-93af-996a4a53a62a","Type":"ContainerStarted","Data":"173b822715915167cb38f5898a576822d50a8432396ff4e7178e73d705541818"} Oct 30 21:47:02.866261 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:02.866201 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" podStartSLOduration=2.866186287 podStartE2EDuration="2.866186287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:02.86266366 +0000 UTC m=+1998.417363447" watchObservedRunningTime="2026-10-30 21:47:02.866186287 +0000 UTC m=+1998.420886058" Oct 30 21:47:03.526365 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:03.524714 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" Oct 30 21:47:03.546382 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:03.542802 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" Oct 30 21:47:03.702652 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:03.702625 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" Oct 30 21:47:03.703527 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:03.703507 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" Oct 30 21:47:11.872361 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:11.870404 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-364e1518-8c6bd84bc-ffns9"] Oct 30 21:47:11.948361 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:11.947751 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-364e1518-8c6bd84bc-ffns9"] Oct 30 21:47:11.948361 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:11.947894 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-364e1518-8c6bd84bc-ffns9" Oct 30 21:47:11.966775 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:11.961738 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-364e1518-655d99d7d-5fmrs"] Oct 30 21:47:12.059363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:12.058186 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-364e1518-dockercfg-jtwqd\"" Oct 30 21:47:12.059363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:12.058292 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-364e1518\"" Oct 30 21:47:12.173417 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:12.120123 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsx6v\" (UniqueName: \"kubernetes.io/projected/3fdf9c46-b0e6-409e-b102-81800c96e24c-kube-api-access-tsx6v\") pod \"payload-pre-processing-e2e-mt-364e1518-8c6bd84bc-ffns9\" (UID: \"3fdf9c46-b0e6-409e-b102-81800c96e24c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-364e1518-8c6bd84bc-ffns9" Oct 30 21:47:12.173976 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:12.173939 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fdf9c46-b0e6-409e-b102-81800c96e24c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-364e1518-8c6bd84bc-ffns9\" (UID: \"3fdf9c46-b0e6-409e-b102-81800c96e24c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-364e1518-8c6bd84bc-ffns9" Oct 30 21:47:12.235698 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:12.235664 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-364e1518-655d99d7d-5fmrs"] Oct 30 21:47:12.235892 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:12.235779 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-364e1518-655d99d7d-5fmrs" Oct 30 21:47:12.275774 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:12.275747 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc1d2d01-9d75-4e02-820d-19d1cbaa18c9-plugins-config-volume\") pod \"payload-processing-e2e-mt-364e1518-655d99d7d-5fmrs\" (UID: \"dc1d2d01-9d75-4e02-820d-19d1cbaa18c9\") " pod="openshift-ingress/payload-processing-e2e-mt-364e1518-655d99d7d-5fmrs" Oct 30 21:47:12.275774 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:12.275778 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4cfz\" (UniqueName: \"kubernetes.io/projected/dc1d2d01-9d75-4e02-820d-19d1cbaa18c9-kube-api-access-k4cfz\") pod \"payload-processing-e2e-mt-364e1518-655d99d7d-5fmrs\" (UID: \"dc1d2d01-9d75-4e02-820d-19d1cbaa18c9\") " pod="openshift-ingress/payload-processing-e2e-mt-364e1518-655d99d7d-5fmrs" Oct 30 21:47:12.275962 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:12.275813 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fdf9c46-b0e6-409e-b102-81800c96e24c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-364e1518-8c6bd84bc-ffns9\" (UID: \"3fdf9c46-b0e6-409e-b102-81800c96e24c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-364e1518-8c6bd84bc-ffns9" Oct 30 21:47:12.276012 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:12.275972 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsx6v\" (UniqueName: \"kubernetes.io/projected/3fdf9c46-b0e6-409e-b102-81800c96e24c-kube-api-access-tsx6v\") pod \"payload-pre-processing-e2e-mt-364e1518-8c6bd84bc-ffns9\" (UID: \"3fdf9c46-b0e6-409e-b102-81800c96e24c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-364e1518-8c6bd84bc-ffns9" Oct 30 21:47:12.276467 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:12.276449 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fdf9c46-b0e6-409e-b102-81800c96e24c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-364e1518-8c6bd84bc-ffns9\" (UID: \"3fdf9c46-b0e6-409e-b102-81800c96e24c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-364e1518-8c6bd84bc-ffns9" Oct 30 21:47:12.334229 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:12.334208 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsx6v\" (UniqueName: \"kubernetes.io/projected/3fdf9c46-b0e6-409e-b102-81800c96e24c-kube-api-access-tsx6v\") pod \"payload-pre-processing-e2e-mt-364e1518-8c6bd84bc-ffns9\" (UID: \"3fdf9c46-b0e6-409e-b102-81800c96e24c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-364e1518-8c6bd84bc-ffns9" Oct 30 21:47:12.377098 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:12.377066 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc1d2d01-9d75-4e02-820d-19d1cbaa18c9-plugins-config-volume\") pod \"payload-processing-e2e-mt-364e1518-655d99d7d-5fmrs\" (UID: \"dc1d2d01-9d75-4e02-820d-19d1cbaa18c9\") " pod="openshift-ingress/payload-processing-e2e-mt-364e1518-655d99d7d-5fmrs" Oct 30 21:47:12.377210 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:12.377103 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4cfz\" (UniqueName: \"kubernetes.io/projected/dc1d2d01-9d75-4e02-820d-19d1cbaa18c9-kube-api-access-k4cfz\") pod \"payload-processing-e2e-mt-364e1518-655d99d7d-5fmrs\" (UID: \"dc1d2d01-9d75-4e02-820d-19d1cbaa18c9\") " pod="openshift-ingress/payload-processing-e2e-mt-364e1518-655d99d7d-5fmrs" Oct 30 21:47:12.377724 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:12.377707 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc1d2d01-9d75-4e02-820d-19d1cbaa18c9-plugins-config-volume\") pod \"payload-processing-e2e-mt-364e1518-655d99d7d-5fmrs\" (UID: \"dc1d2d01-9d75-4e02-820d-19d1cbaa18c9\") " pod="openshift-ingress/payload-processing-e2e-mt-364e1518-655d99d7d-5fmrs" Oct 30 21:47:12.435430 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:12.435382 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4cfz\" (UniqueName: \"kubernetes.io/projected/dc1d2d01-9d75-4e02-820d-19d1cbaa18c9-kube-api-access-k4cfz\") pod \"payload-processing-e2e-mt-364e1518-655d99d7d-5fmrs\" (UID: \"dc1d2d01-9d75-4e02-820d-19d1cbaa18c9\") " pod="openshift-ingress/payload-processing-e2e-mt-364e1518-655d99d7d-5fmrs" Oct 30 21:47:12.560826 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:12.560795 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-364e1518-8c6bd84bc-ffns9" Oct 30 21:47:12.564120 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:12.563773 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-364e1518-655d99d7d-5fmrs" Oct 30 21:47:12.943651 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:12.943621 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-364e1518-8c6bd84bc-ffns9"] Oct 30 21:47:12.954533 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:47:12.954493 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fdf9c46_b0e6_409e_b102_81800c96e24c.slice/crio-88ecafeda0c58ed3af5204f66d9556004d397db22cd20e56838e0682af444e9b WatchSource:0}: Error finding container 88ecafeda0c58ed3af5204f66d9556004d397db22cd20e56838e0682af444e9b: Status 404 returned error can't find the container with id 88ecafeda0c58ed3af5204f66d9556004d397db22cd20e56838e0682af444e9b Oct 30 21:47:12.998397 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:12.998366 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-364e1518-655d99d7d-5fmrs"] Oct 30 21:47:12.999817 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:47:12.999790 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc1d2d01_9d75_4e02_820d_19d1cbaa18c9.slice/crio-b4425e8467cb83f41ae3bfb28511ac462c5e919cc5f7fe5f2c952a55fe6d9da3 WatchSource:0}: Error finding container b4425e8467cb83f41ae3bfb28511ac462c5e919cc5f7fe5f2c952a55fe6d9da3: Status 404 returned error can't find the container with id b4425e8467cb83f41ae3bfb28511ac462c5e919cc5f7fe5f2c952a55fe6d9da3 Oct 30 21:47:13.860557 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:13.860524 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-364e1518-655d99d7d-5fmrs" event={"ID":"dc1d2d01-9d75-4e02-820d-19d1cbaa18c9","Type":"ContainerStarted","Data":"e2691544660f173bea2a847c2846f4eedced4bc846a13e0248f085e0396e192e"} Oct 30 21:47:13.860732 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:13.860580 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-364e1518-655d99d7d-5fmrs" event={"ID":"dc1d2d01-9d75-4e02-820d-19d1cbaa18c9","Type":"ContainerStarted","Data":"b4425e8467cb83f41ae3bfb28511ac462c5e919cc5f7fe5f2c952a55fe6d9da3"} Oct 30 21:47:13.860732 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:13.860609 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-364e1518-655d99d7d-5fmrs" Oct 30 21:47:13.862318 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:13.862295 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-364e1518-8c6bd84bc-ffns9" event={"ID":"3fdf9c46-b0e6-409e-b102-81800c96e24c","Type":"ContainerStarted","Data":"ea127caeac8bed97eaf236c3b367699c02fe50e539a6f805c7b8df2648d3cf4f"} Oct 30 21:47:13.862446 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:13.862348 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-364e1518-8c6bd84bc-ffns9" Oct 30 21:47:13.862446 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:13.862365 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-364e1518-8c6bd84bc-ffns9" event={"ID":"3fdf9c46-b0e6-409e-b102-81800c96e24c","Type":"ContainerStarted","Data":"88ecafeda0c58ed3af5204f66d9556004d397db22cd20e56838e0682af444e9b"} Oct 30 21:47:13.982869 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:13.982820 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-364e1518-655d99d7d-5fmrs" podStartSLOduration=2.98280566 podStartE2EDuration="2.98280566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:13.979362828 +0000 UTC m=+2009.534062604" watchObservedRunningTime="2026-10-30 21:47:13.98280566 +0000 UTC m=+2009.537505436" Oct 30 21:47:14.090131 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:14.090075 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-364e1518-8c6bd84bc-ffns9" podStartSLOduration=3.090056697 podStartE2EDuration="3.090056697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:14.089068481 +0000 UTC m=+2009.643768259" watchObservedRunningTime="2026-10-30 21:47:14.090056697 +0000 UTC m=+2009.644756472" Oct 30 21:47:14.762018 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:14.761988 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:47:16.984364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:16.974241 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" podUID="0ff8ae26-4364-4c4e-9285-721d1a486a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:17.196364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:17.196122 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" podUID="e974bae1-bb18-46b0-a4ee-4c01d6455e32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:17.214091 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:17.214040 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" podUID="8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:19.684256 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:47:19.684216 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:47:24.893761 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:24.893725 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-364e1518-655d99d7d-5fmrs" Oct 30 21:47:24.899263 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:24.899231 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-364e1518-8c6bd84bc-ffns9" Oct 30 21:47:45.884333 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:45.884299 4560 scope.go:122] "RemoveContainer" containerID="7462a7ee02990109b9793d64bed33580a6a1082f6f9128a072b4f71f2529922f" Oct 30 21:47:45.894205 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:45.894185 4560 scope.go:122] "RemoveContainer" containerID="f9f8868087cd2f85910460949156d3ebc246eb37b05a763dd3263d1a633e39f7" Oct 30 21:47:45.903937 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:45.903913 4560 scope.go:122] "RemoveContainer" containerID="3eb16b08f572fc0e9a9e15340c1fd25802097a1d77c934f00deee00840b00382" Oct 30 21:47:45.912464 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:45.912444 4560 scope.go:122] "RemoveContainer" containerID="e207886496bf336adc77ce2b8d8fae1216f98e383e89af316c7c0b0d05cd8ad3" Oct 30 21:47:49.105463 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:49.105430 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc"] Oct 30 21:47:49.111074 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:49.111049 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" Oct 30 21:47:49.207904 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:49.207860 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc"] Oct 30 21:47:49.259684 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:49.259646 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e40614-67ae-43f6-b284-6f09fbc31b08-credential-socket\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc\" (UID: \"16e40614-67ae-43f6-b284-6f09fbc31b08\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" Oct 30 21:47:49.259862 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:49.259701 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16e40614-67ae-43f6-b284-6f09fbc31b08-workload-certs\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc\" (UID: \"16e40614-67ae-43f6-b284-6f09fbc31b08\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" Oct 30 21:47:49.259862 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:49.259850 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16e40614-67ae-43f6-b284-6f09fbc31b08-istio-podinfo\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc\" (UID: \"16e40614-67ae-43f6-b284-6f09fbc31b08\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" Oct 30 21:47:49.259949 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:49.259880 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16e40614-67ae-43f6-b284-6f09fbc31b08-istio-data\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc\" (UID: \"16e40614-67ae-43f6-b284-6f09fbc31b08\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" Oct 30 21:47:49.259949 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:49.259898 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16e40614-67ae-43f6-b284-6f09fbc31b08-istiod-ca-cert\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc\" (UID: \"16e40614-67ae-43f6-b284-6f09fbc31b08\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" Oct 30 21:47:49.259949 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:49.259927 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e40614-67ae-43f6-b284-6f09fbc31b08-workload-socket\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc\" (UID: \"16e40614-67ae-43f6-b284-6f09fbc31b08\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" Oct 30 21:47:49.260166 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:49.260143 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16e40614-67ae-43f6-b284-6f09fbc31b08-istio-token\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc\" (UID: \"16e40614-67ae-43f6-b284-6f09fbc31b08\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" Oct 30 21:47:49.260262 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:49.260244 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8d5z\" (UniqueName: \"kubernetes.io/projected/16e40614-67ae-43f6-b284-6f09fbc31b08-kube-api-access-q8d5z\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc\" (UID: \"16e40614-67ae-43f6-b284-6f09fbc31b08\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" Oct 30 21:47:49.260324 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:49.260309 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16e40614-67ae-43f6-b284-6f09fbc31b08-istio-envoy\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc\" (UID: \"16e40614-67ae-43f6-b284-6f09fbc31b08\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" Oct 30 21:47:49.361099 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:49.361029 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e40614-67ae-43f6-b284-6f09fbc31b08-workload-socket\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc\" (UID: \"16e40614-67ae-43f6-b284-6f09fbc31b08\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" Oct 30 21:47:49.361247 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:49.361105 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16e40614-67ae-43f6-b284-6f09fbc31b08-istio-token\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc\" (UID: \"16e40614-67ae-43f6-b284-6f09fbc31b08\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" Oct 30 21:47:49.361247 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:49.361140 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8d5z\" (UniqueName: \"kubernetes.io/projected/16e40614-67ae-43f6-b284-6f09fbc31b08-kube-api-access-q8d5z\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc\" (UID: \"16e40614-67ae-43f6-b284-6f09fbc31b08\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" Oct 30 21:47:49.361247 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:49.361173 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16e40614-67ae-43f6-b284-6f09fbc31b08-istio-envoy\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc\" (UID: \"16e40614-67ae-43f6-b284-6f09fbc31b08\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" Oct 30 21:47:49.361247 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:49.361233 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e40614-67ae-43f6-b284-6f09fbc31b08-credential-socket\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc\" (UID: \"16e40614-67ae-43f6-b284-6f09fbc31b08\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" Oct 30 21:47:49.361509 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:49.361265 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16e40614-67ae-43f6-b284-6f09fbc31b08-workload-certs\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc\" (UID: \"16e40614-67ae-43f6-b284-6f09fbc31b08\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" Oct 30 21:47:49.361509 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:49.361366 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16e40614-67ae-43f6-b284-6f09fbc31b08-istio-podinfo\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc\" (UID: \"16e40614-67ae-43f6-b284-6f09fbc31b08\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" Oct 30 21:47:49.361509 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:49.361398 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16e40614-67ae-43f6-b284-6f09fbc31b08-istio-data\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc\" (UID: \"16e40614-67ae-43f6-b284-6f09fbc31b08\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" Oct 30 21:47:49.361509 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:49.361419 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16e40614-67ae-43f6-b284-6f09fbc31b08-istiod-ca-cert\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc\" (UID: \"16e40614-67ae-43f6-b284-6f09fbc31b08\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" Oct 30 21:47:49.361699 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:49.361503 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e40614-67ae-43f6-b284-6f09fbc31b08-workload-socket\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc\" (UID: \"16e40614-67ae-43f6-b284-6f09fbc31b08\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" Oct 30 21:47:49.361699 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:49.361621 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e40614-67ae-43f6-b284-6f09fbc31b08-credential-socket\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc\" (UID: \"16e40614-67ae-43f6-b284-6f09fbc31b08\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" Oct 30 21:47:49.361832 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:49.361811 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16e40614-67ae-43f6-b284-6f09fbc31b08-istio-data\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc\" (UID: \"16e40614-67ae-43f6-b284-6f09fbc31b08\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" Oct 30 21:47:49.361884 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:49.361830 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16e40614-67ae-43f6-b284-6f09fbc31b08-workload-certs\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc\" (UID: \"16e40614-67ae-43f6-b284-6f09fbc31b08\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" Oct 30 21:47:49.362081 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:49.362064 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16e40614-67ae-43f6-b284-6f09fbc31b08-istiod-ca-cert\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc\" (UID: \"16e40614-67ae-43f6-b284-6f09fbc31b08\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" Oct 30 21:47:49.363461 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:49.363443 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16e40614-67ae-43f6-b284-6f09fbc31b08-istio-envoy\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc\" (UID: \"16e40614-67ae-43f6-b284-6f09fbc31b08\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" Oct 30 21:47:49.363708 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:49.363689 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16e40614-67ae-43f6-b284-6f09fbc31b08-istio-podinfo\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc\" (UID: \"16e40614-67ae-43f6-b284-6f09fbc31b08\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" Oct 30 21:47:49.419159 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:49.419129 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16e40614-67ae-43f6-b284-6f09fbc31b08-istio-token\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc\" (UID: \"16e40614-67ae-43f6-b284-6f09fbc31b08\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" Oct 30 21:47:49.419267 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:49.419188 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8d5z\" (UniqueName: \"kubernetes.io/projected/16e40614-67ae-43f6-b284-6f09fbc31b08-kube-api-access-q8d5z\") pod \"e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc\" (UID: \"16e40614-67ae-43f6-b284-6f09fbc31b08\") " pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" Oct 30 21:47:49.424018 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:49.423994 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" Oct 30 21:47:49.686465 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:47:49.686385 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:47:49.863210 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:49.863184 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc"] Oct 30 21:47:49.866369 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:47:49.866331 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e40614_67ae_43f6_b284_6f09fbc31b08.slice/crio-264ea75e6f7429cb87af50c80f5543abc787abbfdd3982c0c9e685bfeb9ba52d WatchSource:0}: Error finding container 264ea75e6f7429cb87af50c80f5543abc787abbfdd3982c0c9e685bfeb9ba52d: Status 404 returned error can't find the container with id 264ea75e6f7429cb87af50c80f5543abc787abbfdd3982c0c9e685bfeb9ba52d Oct 30 21:47:49.868833 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:49.868799 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:47:49.868918 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:49.868866 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:47:49.868918 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:49.868893 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:47:50.187074 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:50.187034 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" event={"ID":"16e40614-67ae-43f6-b284-6f09fbc31b08","Type":"ContainerStarted","Data":"96498f510a6c123d37b3f0d531bfe6eca88e78257462d40596850ac567f50104"} Oct 30 21:47:50.187495 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:50.187121 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" event={"ID":"16e40614-67ae-43f6-b284-6f09fbc31b08","Type":"ContainerStarted","Data":"264ea75e6f7429cb87af50c80f5543abc787abbfdd3982c0c9e685bfeb9ba52d"} Oct 30 21:47:50.304558 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:50.304496 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" podStartSLOduration=1.304479507 podStartE2EDuration="1.304479507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:50.301277348 +0000 UTC m=+2045.855977124" watchObservedRunningTime="2026-10-30 21:47:50.304479507 +0000 UTC m=+2045.859179285" Oct 30 21:47:50.425244 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:50.425203 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" Oct 30 21:47:50.437432 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:50.436466 4560 patch_prober.go:29] interesting pod/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 21:47:50.437432 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:50.436535 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" podUID="16e40614-67ae-43f6-b284-6f09fbc31b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 21:47:51.424870 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:51.424836 4560 patch_prober.go:29] interesting pod/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 21:47:51.425286 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:51.424896 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" podUID="16e40614-67ae-43f6-b284-6f09fbc31b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 21:47:52.425006 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:52.424967 4560 patch_prober.go:29] interesting pod/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 21:47:52.425522 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:52.425028 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" podUID="16e40614-67ae-43f6-b284-6f09fbc31b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 21:47:53.425180 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:53.425144 4560 patch_prober.go:29] interesting pod/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 21:47:53.425702 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:53.425213 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" podUID="16e40614-67ae-43f6-b284-6f09fbc31b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 21:47:54.440363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:54.428766 4560 patch_prober.go:29] interesting pod/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 21:47:54.440363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:54.428834 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" podUID="16e40614-67ae-43f6-b284-6f09fbc31b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 21:47:55.429900 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:55.429860 4560 patch_prober.go:29] interesting pod/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 21:47:55.430211 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:55.430174 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" podUID="16e40614-67ae-43f6-b284-6f09fbc31b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 21:47:56.425278 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:56.425229 4560 patch_prober.go:29] interesting pod/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 21:47:56.425678 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:56.425308 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" podUID="16e40614-67ae-43f6-b284-6f09fbc31b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 21:47:57.430069 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:57.430041 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" Oct 30 21:47:57.430519 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:57.430430 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" Oct 30 21:47:57.431076 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:47:57.431060 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" Oct 30 21:48:01.075108 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:01.075060 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:48:01.207897 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:01.200332 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e79f61de-4mxt7"] Oct 30 21:48:01.238760 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:01.238666 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e79f61de-4mxt7" Oct 30 21:48:01.301850 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:01.301811 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e79f61de-4mxt7"] Oct 30 21:48:01.363422 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:01.362404 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:48:01.363422 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:01.362565 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:48:01.363422 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:01.362649 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:48:01.363422 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:01.363117 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzfqg\" (UniqueName: \"kubernetes.io/projected/1865ffda-3cf3-4f84-b111-d417ce2de3ec-kube-api-access-hzfqg\") pod \"maas-api-revoke-keys-e2e-ait-e79f61de-4mxt7\" (UID: \"1865ffda-3cf3-4f84-b111-d417ce2de3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e79f61de-4mxt7" Oct 30 21:48:01.363422 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:01.363210 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:48:01.363422 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:01.363321 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1865ffda-3cf3-4f84-b111-d417ce2de3ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e79f61de-4mxt7\" (UID: \"1865ffda-3cf3-4f84-b111-d417ce2de3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e79f61de-4mxt7" Oct 30 21:48:01.363905 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:01.363564 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1865ffda-3cf3-4f84-b111-d417ce2de3ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e79f61de-4mxt7\" (UID: \"1865ffda-3cf3-4f84-b111-d417ce2de3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e79f61de-4mxt7" Oct 30 21:48:01.465280 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:01.465224 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1865ffda-3cf3-4f84-b111-d417ce2de3ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e79f61de-4mxt7\" (UID: \"1865ffda-3cf3-4f84-b111-d417ce2de3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e79f61de-4mxt7" Oct 30 21:48:01.465496 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:01.465412 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1865ffda-3cf3-4f84-b111-d417ce2de3ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e79f61de-4mxt7\" (UID: \"1865ffda-3cf3-4f84-b111-d417ce2de3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e79f61de-4mxt7" Oct 30 21:48:01.465580 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:01.465565 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzfqg\" (UniqueName: \"kubernetes.io/projected/1865ffda-3cf3-4f84-b111-d417ce2de3ec-kube-api-access-hzfqg\") pod \"maas-api-revoke-keys-e2e-ait-e79f61de-4mxt7\" (UID: \"1865ffda-3cf3-4f84-b111-d417ce2de3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e79f61de-4mxt7" Oct 30 21:48:01.466222 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:01.466195 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1865ffda-3cf3-4f84-b111-d417ce2de3ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e79f61de-4mxt7\" (UID: \"1865ffda-3cf3-4f84-b111-d417ce2de3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e79f61de-4mxt7" Oct 30 21:48:01.468018 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:01.467989 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1865ffda-3cf3-4f84-b111-d417ce2de3ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e79f61de-4mxt7\" (UID: \"1865ffda-3cf3-4f84-b111-d417ce2de3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e79f61de-4mxt7" Oct 30 21:48:01.524059 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:01.524012 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzfqg\" (UniqueName: \"kubernetes.io/projected/1865ffda-3cf3-4f84-b111-d417ce2de3ec-kube-api-access-hzfqg\") pod \"maas-api-revoke-keys-e2e-ait-e79f61de-4mxt7\" (UID: \"1865ffda-3cf3-4f84-b111-d417ce2de3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e79f61de-4mxt7" Oct 30 21:48:01.659562 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:01.659489 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e79f61de-4mxt7" Oct 30 21:48:02.379226 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:02.379195 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e79f61de-4mxt7"] Oct 30 21:48:02.383133 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:48:02.383100 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1865ffda_3cf3_4f84_b111_d417ce2de3ec.slice/crio-40ffd84cfcf5c0d0c80e901b02e596e51b1f6f57970683424e3f2118a261f7f5 WatchSource:0}: Error finding container 40ffd84cfcf5c0d0c80e901b02e596e51b1f6f57970683424e3f2118a261f7f5: Status 404 returned error can't find the container with id 40ffd84cfcf5c0d0c80e901b02e596e51b1f6f57970683424e3f2118a261f7f5 Oct 30 21:48:02.414377 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:02.414355 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e79f61de-4mxt7" event={"ID":"1865ffda-3cf3-4f84-b111-d417ce2de3ec","Type":"ContainerStarted","Data":"40ffd84cfcf5c0d0c80e901b02e596e51b1f6f57970683424e3f2118a261f7f5"} Oct 30 21:48:03.422726 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:03.422691 4560 generic.go:356] "Generic (PLEG): container finished" podID="1865ffda-3cf3-4f84-b111-d417ce2de3ec" containerID="67419d7320bc500ce260826f8212a3a007520729385f8a8ca31f4a58260a2976" exitCode=0 Oct 30 21:48:03.423091 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:03.422777 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e79f61de-4mxt7" event={"ID":"1865ffda-3cf3-4f84-b111-d417ce2de3ec","Type":"ContainerDied","Data":"67419d7320bc500ce260826f8212a3a007520729385f8a8ca31f4a58260a2976"} Oct 30 21:48:04.605501 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:04.605477 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e79f61de-4mxt7" Oct 30 21:48:04.684178 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:04.684088 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1865ffda-3cf3-4f84-b111-d417ce2de3ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1865ffda-3cf3-4f84-b111-d417ce2de3ec-maas-api-openshift-ca\") pod \"1865ffda-3cf3-4f84-b111-d417ce2de3ec\" (UID: \"1865ffda-3cf3-4f84-b111-d417ce2de3ec\") " Oct 30 21:48:04.684534 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:04.684502 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1865ffda-3cf3-4f84-b111-d417ce2de3ec-maas-api-openshift-ca" pod "1865ffda-3cf3-4f84-b111-d417ce2de3ec" (UID: "1865ffda-3cf3-4f84-b111-d417ce2de3ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:04.684651 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:04.684585 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1865ffda-3cf3-4f84-b111-d417ce2de3ec-kube-api-access-hzfqg\" (UniqueName: \"kubernetes.io/projected/1865ffda-3cf3-4f84-b111-d417ce2de3ec-kube-api-access-hzfqg\") pod \"1865ffda-3cf3-4f84-b111-d417ce2de3ec\" (UID: \"1865ffda-3cf3-4f84-b111-d417ce2de3ec\") " Oct 30 21:48:04.684747 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:04.684666 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1865ffda-3cf3-4f84-b111-d417ce2de3ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1865ffda-3cf3-4f84-b111-d417ce2de3ec-maas-api-xks-ca\") pod \"1865ffda-3cf3-4f84-b111-d417ce2de3ec\" (UID: \"1865ffda-3cf3-4f84-b111-d417ce2de3ec\") " Oct 30 21:48:04.685451 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:04.685410 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1865ffda-3cf3-4f84-b111-d417ce2de3ec-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:04.687633 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:04.687589 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1865ffda-3cf3-4f84-b111-d417ce2de3ec-kube-api-access-hzfqg" pod "1865ffda-3cf3-4f84-b111-d417ce2de3ec" (UID: "1865ffda-3cf3-4f84-b111-d417ce2de3ec"). InnerVolumeSpecName "kube-api-access-hzfqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:04.690256 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:04.689912 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1865ffda-3cf3-4f84-b111-d417ce2de3ec-maas-api-xks-ca" pod "1865ffda-3cf3-4f84-b111-d417ce2de3ec" (UID: "1865ffda-3cf3-4f84-b111-d417ce2de3ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:04.786881 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:04.786855 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzfqg\" (UniqueName: \"kubernetes.io/projected/1865ffda-3cf3-4f84-b111-d417ce2de3ec-kube-api-access-hzfqg\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:04.786881 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:04.786882 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1865ffda-3cf3-4f84-b111-d417ce2de3ec-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:05.440735 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:05.440702 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e79f61de-4mxt7" Oct 30 21:48:05.440735 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:05.440709 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e79f61de-4mxt7" event={"ID":"1865ffda-3cf3-4f84-b111-d417ce2de3ec","Type":"ContainerDied","Data":"40ffd84cfcf5c0d0c80e901b02e596e51b1f6f57970683424e3f2118a261f7f5"} Oct 30 21:48:05.440912 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:05.440768 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40ffd84cfcf5c0d0c80e901b02e596e51b1f6f57970683424e3f2118a261f7f5" Oct 30 21:48:12.678363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:12.671489 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a84c1592-4swx2"] Oct 30 21:48:12.678363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:12.672554 4560 cpu_manager.go:414] "containerMap: removing container" podUID="1865ffda-3cf3-4f84-b111-d417ce2de3ec" containerName="revoke-keys" Oct 30 21:48:12.678363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:12.672576 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1865ffda-3cf3-4f84-b111-d417ce2de3ec" containerName="revoke-keys" Oct 30 21:48:12.678363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:12.672761 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="1865ffda-3cf3-4f84-b111-d417ce2de3ec" containerName="revoke-keys" Oct 30 21:48:12.818302 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:12.818264 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a84c1592-4swx2"] Oct 30 21:48:12.818675 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:12.818435 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a84c1592-4swx2" Oct 30 21:48:12.886360 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:12.886217 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:48:12.894202 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:12.891901 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:48:12.894202 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:12.892013 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:48:12.932828 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:12.932760 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:48:12.956035 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:12.955997 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da33cb18-0fb2-47ee-a685-c2f1a72bd21e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a84c1592-4swx2\" (UID: \"da33cb18-0fb2-47ee-a685-c2f1a72bd21e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a84c1592-4swx2" Oct 30 21:48:12.956199 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:12.956174 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb8ch\" (UniqueName: \"kubernetes.io/projected/da33cb18-0fb2-47ee-a685-c2f1a72bd21e-kube-api-access-mb8ch\") pod \"maas-api-revoke-keys-e2e-ait-a84c1592-4swx2\" (UID: \"da33cb18-0fb2-47ee-a685-c2f1a72bd21e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a84c1592-4swx2" Oct 30 21:48:12.956306 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:12.956287 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da33cb18-0fb2-47ee-a685-c2f1a72bd21e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a84c1592-4swx2\" (UID: \"da33cb18-0fb2-47ee-a685-c2f1a72bd21e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a84c1592-4swx2" Oct 30 21:48:13.058019 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:13.057986 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb8ch\" (UniqueName: \"kubernetes.io/projected/da33cb18-0fb2-47ee-a685-c2f1a72bd21e-kube-api-access-mb8ch\") pod \"maas-api-revoke-keys-e2e-ait-a84c1592-4swx2\" (UID: \"da33cb18-0fb2-47ee-a685-c2f1a72bd21e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a84c1592-4swx2" Oct 30 21:48:13.058166 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:13.058075 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da33cb18-0fb2-47ee-a685-c2f1a72bd21e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a84c1592-4swx2\" (UID: \"da33cb18-0fb2-47ee-a685-c2f1a72bd21e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a84c1592-4swx2" Oct 30 21:48:13.058206 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:13.058174 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da33cb18-0fb2-47ee-a685-c2f1a72bd21e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a84c1592-4swx2\" (UID: \"da33cb18-0fb2-47ee-a685-c2f1a72bd21e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a84c1592-4swx2" Oct 30 21:48:13.058813 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:13.058795 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da33cb18-0fb2-47ee-a685-c2f1a72bd21e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a84c1592-4swx2\" (UID: \"da33cb18-0fb2-47ee-a685-c2f1a72bd21e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a84c1592-4swx2" Oct 30 21:48:13.060430 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:13.060409 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da33cb18-0fb2-47ee-a685-c2f1a72bd21e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a84c1592-4swx2\" (UID: \"da33cb18-0fb2-47ee-a685-c2f1a72bd21e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a84c1592-4swx2" Oct 30 21:48:13.116070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:13.116021 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb8ch\" (UniqueName: \"kubernetes.io/projected/da33cb18-0fb2-47ee-a685-c2f1a72bd21e-kube-api-access-mb8ch\") pod \"maas-api-revoke-keys-e2e-ait-a84c1592-4swx2\" (UID: \"da33cb18-0fb2-47ee-a685-c2f1a72bd21e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a84c1592-4swx2" Oct 30 21:48:13.195702 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:13.195639 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a84c1592-4swx2" Oct 30 21:48:13.575160 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:13.575131 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a84c1592-4swx2"] Oct 30 21:48:13.578843 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:48:13.578816 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda33cb18_0fb2_47ee_a685_c2f1a72bd21e.slice/crio-dfa9a028fcf48efb45b8679a8b902ecdbb2887f0c3e3b7551aa932b835734e14 WatchSource:0}: Error finding container dfa9a028fcf48efb45b8679a8b902ecdbb2887f0c3e3b7551aa932b835734e14: Status 404 returned error can't find the container with id dfa9a028fcf48efb45b8679a8b902ecdbb2887f0c3e3b7551aa932b835734e14 Oct 30 21:48:14.506444 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:14.506405 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a84c1592-4swx2" event={"ID":"da33cb18-0fb2-47ee-a685-c2f1a72bd21e","Type":"ContainerStarted","Data":"1cbd9e656ba2581911305c01a5a7e96ce581fe04a37cab0377361c098733a00a"} Oct 30 21:48:14.506856 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:14.506465 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a84c1592-4swx2" event={"ID":"da33cb18-0fb2-47ee-a685-c2f1a72bd21e","Type":"ContainerStarted","Data":"dfa9a028fcf48efb45b8679a8b902ecdbb2887f0c3e3b7551aa932b835734e14"} Oct 30 21:48:14.630558 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:14.630506 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a84c1592-4swx2" podStartSLOduration=2.630492806 podStartE2EDuration="2.630492806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:14.623453537 +0000 UTC m=+2070.178153314" watchObservedRunningTime="2026-10-30 21:48:14.630492806 +0000 UTC m=+2070.185192581" Oct 30 21:48:15.514204 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:15.514170 4560 generic.go:356] "Generic (PLEG): container finished" podID="da33cb18-0fb2-47ee-a685-c2f1a72bd21e" containerID="1cbd9e656ba2581911305c01a5a7e96ce581fe04a37cab0377361c098733a00a" exitCode=0 Oct 30 21:48:15.514655 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:15.514254 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a84c1592-4swx2" event={"ID":"da33cb18-0fb2-47ee-a685-c2f1a72bd21e","Type":"ContainerDied","Data":"1cbd9e656ba2581911305c01a5a7e96ce581fe04a37cab0377361c098733a00a"} Oct 30 21:48:16.788539 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:16.788483 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a84c1592-4swx2" Oct 30 21:48:16.921350 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:16.921301 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da33cb18-0fb2-47ee-a685-c2f1a72bd21e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da33cb18-0fb2-47ee-a685-c2f1a72bd21e-maas-api-xks-ca\") pod \"da33cb18-0fb2-47ee-a685-c2f1a72bd21e\" (UID: \"da33cb18-0fb2-47ee-a685-c2f1a72bd21e\") " Oct 30 21:48:16.921521 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:16.921388 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da33cb18-0fb2-47ee-a685-c2f1a72bd21e-kube-api-access-mb8ch\" (UniqueName: \"kubernetes.io/projected/da33cb18-0fb2-47ee-a685-c2f1a72bd21e-kube-api-access-mb8ch\") pod \"da33cb18-0fb2-47ee-a685-c2f1a72bd21e\" (UID: \"da33cb18-0fb2-47ee-a685-c2f1a72bd21e\") " Oct 30 21:48:16.921614 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:16.921593 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da33cb18-0fb2-47ee-a685-c2f1a72bd21e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da33cb18-0fb2-47ee-a685-c2f1a72bd21e-maas-api-openshift-ca\") pod \"da33cb18-0fb2-47ee-a685-c2f1a72bd21e\" (UID: \"da33cb18-0fb2-47ee-a685-c2f1a72bd21e\") " Oct 30 21:48:16.922127 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:16.922097 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da33cb18-0fb2-47ee-a685-c2f1a72bd21e-maas-api-openshift-ca" pod "da33cb18-0fb2-47ee-a685-c2f1a72bd21e" (UID: "da33cb18-0fb2-47ee-a685-c2f1a72bd21e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:16.924002 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:16.923980 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da33cb18-0fb2-47ee-a685-c2f1a72bd21e-maas-api-xks-ca" pod "da33cb18-0fb2-47ee-a685-c2f1a72bd21e" (UID: "da33cb18-0fb2-47ee-a685-c2f1a72bd21e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:16.924398 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:16.924377 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da33cb18-0fb2-47ee-a685-c2f1a72bd21e-kube-api-access-mb8ch" pod "da33cb18-0fb2-47ee-a685-c2f1a72bd21e" (UID: "da33cb18-0fb2-47ee-a685-c2f1a72bd21e"). InnerVolumeSpecName "kube-api-access-mb8ch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:17.023766 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:17.023740 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da33cb18-0fb2-47ee-a685-c2f1a72bd21e-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:17.023766 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:17.023765 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb8ch\" (UniqueName: \"kubernetes.io/projected/da33cb18-0fb2-47ee-a685-c2f1a72bd21e-kube-api-access-mb8ch\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:17.023932 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:17.023777 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da33cb18-0fb2-47ee-a685-c2f1a72bd21e-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:17.528817 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:17.528790 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a84c1592-4swx2" Oct 30 21:48:17.528989 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:17.528819 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a84c1592-4swx2" event={"ID":"da33cb18-0fb2-47ee-a685-c2f1a72bd21e","Type":"ContainerDied","Data":"dfa9a028fcf48efb45b8679a8b902ecdbb2887f0c3e3b7551aa932b835734e14"} Oct 30 21:48:17.528989 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:17.528866 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfa9a028fcf48efb45b8679a8b902ecdbb2887f0c3e3b7551aa932b835734e14" Oct 30 21:48:19.686078 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:48:19.686038 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:48:21.057847 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:21.057812 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0872-v8dcn"] Oct 30 21:48:21.124769 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:21.124716 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0872-v8dcn"] Oct 30 21:48:22.366391 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:22.366359 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-899a6bb1-84m6t"] Oct 30 21:48:22.366910 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:22.366897 4560 cpu_manager.go:414] "containerMap: removing container" podUID="da33cb18-0fb2-47ee-a685-c2f1a72bd21e" containerName="revoke-keys" Oct 30 21:48:22.366910 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:22.366911 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da33cb18-0fb2-47ee-a685-c2f1a72bd21e" containerName="revoke-keys" Oct 30 21:48:22.367014 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:22.367002 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="da33cb18-0fb2-47ee-a685-c2f1a72bd21e" containerName="revoke-keys" Oct 30 21:48:22.370585 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:22.370568 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-899a6bb1-84m6t" Oct 30 21:48:22.423153 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:22.423110 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:48:22.469897 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:22.469868 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:48:22.470024 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:22.469914 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:48:22.470024 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:22.469914 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:48:22.474393 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:22.474373 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-899a6bb1-84m6t"] Oct 30 21:48:22.515771 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:22.515749 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4527511-5f50-4d62-bb3b-f6537457b147-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-899a6bb1-84m6t\" (UID: \"e4527511-5f50-4d62-bb3b-f6537457b147\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-899a6bb1-84m6t" Oct 30 21:48:22.515869 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:22.515799 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4527511-5f50-4d62-bb3b-f6537457b147-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-899a6bb1-84m6t\" (UID: \"e4527511-5f50-4d62-bb3b-f6537457b147\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-899a6bb1-84m6t" Oct 30 21:48:22.516010 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:22.515989 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx984\" (UniqueName: \"kubernetes.io/projected/e4527511-5f50-4d62-bb3b-f6537457b147-kube-api-access-bx984\") pod \"maas-api-revoke-keys-e2e-derive-899a6bb1-84m6t\" (UID: \"e4527511-5f50-4d62-bb3b-f6537457b147\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-899a6bb1-84m6t" Oct 30 21:48:22.575624 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:22.575599 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20465276-05de-4b00-80fa-d40c49a55d0e" path="/var/lib/kubelet/pods/20465276-05de-4b00-80fa-d40c49a55d0e/volumes" Oct 30 21:48:22.617291 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:22.617210 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx984\" (UniqueName: \"kubernetes.io/projected/e4527511-5f50-4d62-bb3b-f6537457b147-kube-api-access-bx984\") pod \"maas-api-revoke-keys-e2e-derive-899a6bb1-84m6t\" (UID: \"e4527511-5f50-4d62-bb3b-f6537457b147\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-899a6bb1-84m6t" Oct 30 21:48:22.617291 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:22.617276 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4527511-5f50-4d62-bb3b-f6537457b147-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-899a6bb1-84m6t\" (UID: \"e4527511-5f50-4d62-bb3b-f6537457b147\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-899a6bb1-84m6t" Oct 30 21:48:22.617542 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:22.617351 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4527511-5f50-4d62-bb3b-f6537457b147-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-899a6bb1-84m6t\" (UID: \"e4527511-5f50-4d62-bb3b-f6537457b147\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-899a6bb1-84m6t" Oct 30 21:48:22.618057 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:22.618033 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4527511-5f50-4d62-bb3b-f6537457b147-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-899a6bb1-84m6t\" (UID: \"e4527511-5f50-4d62-bb3b-f6537457b147\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-899a6bb1-84m6t" Oct 30 21:48:22.619806 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:22.619785 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4527511-5f50-4d62-bb3b-f6537457b147-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-899a6bb1-84m6t\" (UID: \"e4527511-5f50-4d62-bb3b-f6537457b147\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-899a6bb1-84m6t" Oct 30 21:48:22.675691 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:22.675662 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx984\" (UniqueName: \"kubernetes.io/projected/e4527511-5f50-4d62-bb3b-f6537457b147-kube-api-access-bx984\") pod \"maas-api-revoke-keys-e2e-derive-899a6bb1-84m6t\" (UID: \"e4527511-5f50-4d62-bb3b-f6537457b147\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-899a6bb1-84m6t" Oct 30 21:48:22.683456 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:22.683434 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-899a6bb1-84m6t" Oct 30 21:48:23.059869 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:23.059837 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-899a6bb1-84m6t"] Oct 30 21:48:23.063214 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:48:23.063190 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4527511_5f50_4d62_bb3b_f6537457b147.slice/crio-73d328be8daebd46e14af3b58849aa2e6c77f9a94eb1428e252ec7c026c0c3d1 WatchSource:0}: Error finding container 73d328be8daebd46e14af3b58849aa2e6c77f9a94eb1428e252ec7c026c0c3d1: Status 404 returned error can't find the container with id 73d328be8daebd46e14af3b58849aa2e6c77f9a94eb1428e252ec7c026c0c3d1 Oct 30 21:48:23.575428 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:23.575390 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-899a6bb1-84m6t" event={"ID":"e4527511-5f50-4d62-bb3b-f6537457b147","Type":"ContainerStarted","Data":"b89dc3d6be7dd622c04b91ac31852c79e76cdb88d351fd94c2b264a67cde1362"} Oct 30 21:48:23.575852 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:23.575434 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-899a6bb1-84m6t" event={"ID":"e4527511-5f50-4d62-bb3b-f6537457b147","Type":"ContainerStarted","Data":"73d328be8daebd46e14af3b58849aa2e6c77f9a94eb1428e252ec7c026c0c3d1"} Oct 30 21:48:23.689911 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:23.689816 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-899a6bb1-84m6t" podStartSLOduration=1.6898038560000002 podStartE2EDuration="1.689803856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:23.686259876 +0000 UTC m=+2079.240959654" watchObservedRunningTime="2026-10-30 21:48:23.689803856 +0000 UTC m=+2079.244503632" Oct 30 21:48:24.583737 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:24.583705 4560 generic.go:356] "Generic (PLEG): container finished" podID="e4527511-5f50-4d62-bb3b-f6537457b147" containerID="b89dc3d6be7dd622c04b91ac31852c79e76cdb88d351fd94c2b264a67cde1362" exitCode=0 Oct 30 21:48:24.584161 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:24.583785 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-899a6bb1-84m6t" event={"ID":"e4527511-5f50-4d62-bb3b-f6537457b147","Type":"ContainerDied","Data":"b89dc3d6be7dd622c04b91ac31852c79e76cdb88d351fd94c2b264a67cde1362"} Oct 30 21:48:25.897764 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:25.897729 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-899a6bb1-84m6t" Oct 30 21:48:25.941148 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:25.941092 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4527511-5f50-4d62-bb3b-f6537457b147-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4527511-5f50-4d62-bb3b-f6537457b147-maas-api-xks-ca\") pod \"e4527511-5f50-4d62-bb3b-f6537457b147\" (UID: \"e4527511-5f50-4d62-bb3b-f6537457b147\") " Oct 30 21:48:25.941278 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:25.941206 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4527511-5f50-4d62-bb3b-f6537457b147-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4527511-5f50-4d62-bb3b-f6537457b147-maas-api-openshift-ca\") pod \"e4527511-5f50-4d62-bb3b-f6537457b147\" (UID: \"e4527511-5f50-4d62-bb3b-f6537457b147\") " Oct 30 21:48:25.941278 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:25.941266 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4527511-5f50-4d62-bb3b-f6537457b147-kube-api-access-bx984\" (UniqueName: \"kubernetes.io/projected/e4527511-5f50-4d62-bb3b-f6537457b147-kube-api-access-bx984\") pod \"e4527511-5f50-4d62-bb3b-f6537457b147\" (UID: \"e4527511-5f50-4d62-bb3b-f6537457b147\") " Oct 30 21:48:25.941588 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:25.941565 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4527511-5f50-4d62-bb3b-f6537457b147-maas-api-openshift-ca" pod "e4527511-5f50-4d62-bb3b-f6537457b147" (UID: "e4527511-5f50-4d62-bb3b-f6537457b147"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:25.941860 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:25.941843 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4527511-5f50-4d62-bb3b-f6537457b147-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:25.943020 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:25.943001 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4527511-5f50-4d62-bb3b-f6537457b147-maas-api-xks-ca" pod "e4527511-5f50-4d62-bb3b-f6537457b147" (UID: "e4527511-5f50-4d62-bb3b-f6537457b147"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:25.943109 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:25.943089 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4527511-5f50-4d62-bb3b-f6537457b147-kube-api-access-bx984" pod "e4527511-5f50-4d62-bb3b-f6537457b147" (UID: "e4527511-5f50-4d62-bb3b-f6537457b147"). InnerVolumeSpecName "kube-api-access-bx984". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:26.043095 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:26.043066 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4527511-5f50-4d62-bb3b-f6537457b147-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:26.043095 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:26.043091 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx984\" (UniqueName: \"kubernetes.io/projected/e4527511-5f50-4d62-bb3b-f6537457b147-kube-api-access-bx984\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:26.652652 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:26.652621 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-899a6bb1-84m6t" Oct 30 21:48:26.652808 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:26.652658 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-899a6bb1-84m6t" event={"ID":"e4527511-5f50-4d62-bb3b-f6537457b147","Type":"ContainerDied","Data":"73d328be8daebd46e14af3b58849aa2e6c77f9a94eb1428e252ec7c026c0c3d1"} Oct 30 21:48:26.652808 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:26.652705 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73d328be8daebd46e14af3b58849aa2e6c77f9a94eb1428e252ec7c026c0c3d1" Oct 30 21:48:39.872397 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:39.872364 4560 generic.go:356] "Generic (PLEG): container finished" podID="5c0221dd-a19a-464f-93af-996a4a53a62a" containerID="3b3013fa31a125cb8facbd94d31e33324f30129daa8160f1fa385b10c36fbc6e" exitCode=0 Oct 30 21:48:39.872824 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:39.872441 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" event={"ID":"5c0221dd-a19a-464f-93af-996a4a53a62a","Type":"ContainerDied","Data":"3b3013fa31a125cb8facbd94d31e33324f30129daa8160f1fa385b10c36fbc6e"} Oct 30 21:48:39.872824 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:39.872816 4560 scope.go:122] "RemoveContainer" containerID="3b3013fa31a125cb8facbd94d31e33324f30129daa8160f1fa385b10c36fbc6e" Oct 30 21:48:39.874494 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:39.874463 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:48:39.874613 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:39.874537 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:48:39.874613 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:39.874585 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:48:40.881489 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:40.881452 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" event={"ID":"5c0221dd-a19a-464f-93af-996a4a53a62a","Type":"ContainerStarted","Data":"c864772955893be48a5dee27e687d4a2e92e16f3295b72a1874d0a116483e381"} Oct 30 21:48:41.519967 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:41.519937 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" Oct 30 21:48:41.520190 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:41.520148 4560 patch_prober.go:29] interesting pod/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:48:41.520275 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:41.520206 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" podUID="5c0221dd-a19a-464f-93af-996a4a53a62a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 21:48:42.520430 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:42.520397 4560 patch_prober.go:29] interesting pod/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:48:42.520847 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:42.520446 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" podUID="5c0221dd-a19a-464f-93af-996a4a53a62a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 21:48:43.520614 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:43.520572 4560 patch_prober.go:29] interesting pod/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:48:43.521187 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:43.520650 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" podUID="5c0221dd-a19a-464f-93af-996a4a53a62a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 21:48:44.520514 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:44.520478 4560 patch_prober.go:29] interesting pod/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:48:44.520682 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:44.520527 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" podUID="5c0221dd-a19a-464f-93af-996a4a53a62a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 21:48:45.519819 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:45.519782 4560 patch_prober.go:29] interesting pod/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:48:45.519985 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:45.519832 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" podUID="5c0221dd-a19a-464f-93af-996a4a53a62a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 21:48:46.559538 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:46.558667 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" Oct 30 21:48:46.572371 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:46.560211 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" Oct 30 21:48:46.644463 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:46.644431 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" Oct 30 21:48:49.577663 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:49.577630 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af87dbaf-qbvvf"] Oct 30 21:48:49.578417 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:49.578399 4560 cpu_manager.go:414] "containerMap: removing container" podUID="e4527511-5f50-4d62-bb3b-f6537457b147" containerName="revoke-keys" Oct 30 21:48:49.578477 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:49.578419 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4527511-5f50-4d62-bb3b-f6537457b147" containerName="revoke-keys" Oct 30 21:48:49.578562 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:49.578552 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4527511-5f50-4d62-bb3b-f6537457b147" containerName="revoke-keys" Oct 30 21:48:49.582728 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:49.582710 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af87dbaf-qbvvf" Oct 30 21:48:49.634731 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:49.634704 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af87dbaf-qbvvf"] Oct 30 21:48:49.675757 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:49.675725 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:48:49.675992 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:49.675760 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:48:49.675992 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:49.675782 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:48:49.678943 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:49.678919 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:48:49.688114 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:48:49.688084 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:48:49.710899 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:49.710871 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nkln\" (UniqueName: \"kubernetes.io/projected/8df54091-af1d-4a2a-bd25-e1e5068a26be-kube-api-access-6nkln\") pod \"maas-api-revoke-keys-e2e-ait-af87dbaf-qbvvf\" (UID: \"8df54091-af1d-4a2a-bd25-e1e5068a26be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af87dbaf-qbvvf" Oct 30 21:48:49.711180 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:49.711157 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8df54091-af1d-4a2a-bd25-e1e5068a26be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-af87dbaf-qbvvf\" (UID: \"8df54091-af1d-4a2a-bd25-e1e5068a26be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af87dbaf-qbvvf" Oct 30 21:48:49.711390 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:49.711369 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8df54091-af1d-4a2a-bd25-e1e5068a26be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-af87dbaf-qbvvf\" (UID: \"8df54091-af1d-4a2a-bd25-e1e5068a26be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af87dbaf-qbvvf" Oct 30 21:48:49.812250 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:49.812215 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8df54091-af1d-4a2a-bd25-e1e5068a26be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-af87dbaf-qbvvf\" (UID: \"8df54091-af1d-4a2a-bd25-e1e5068a26be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af87dbaf-qbvvf" Oct 30 21:48:49.812445 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:49.812318 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8df54091-af1d-4a2a-bd25-e1e5068a26be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-af87dbaf-qbvvf\" (UID: \"8df54091-af1d-4a2a-bd25-e1e5068a26be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af87dbaf-qbvvf" Oct 30 21:48:49.812445 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:49.812401 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nkln\" (UniqueName: \"kubernetes.io/projected/8df54091-af1d-4a2a-bd25-e1e5068a26be-kube-api-access-6nkln\") pod \"maas-api-revoke-keys-e2e-ait-af87dbaf-qbvvf\" (UID: \"8df54091-af1d-4a2a-bd25-e1e5068a26be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af87dbaf-qbvvf" Oct 30 21:48:49.812870 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:49.812850 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8df54091-af1d-4a2a-bd25-e1e5068a26be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-af87dbaf-qbvvf\" (UID: \"8df54091-af1d-4a2a-bd25-e1e5068a26be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af87dbaf-qbvvf" Oct 30 21:48:49.814663 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:49.814641 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8df54091-af1d-4a2a-bd25-e1e5068a26be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-af87dbaf-qbvvf\" (UID: \"8df54091-af1d-4a2a-bd25-e1e5068a26be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af87dbaf-qbvvf" Oct 30 21:48:49.871008 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:49.870942 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nkln\" (UniqueName: \"kubernetes.io/projected/8df54091-af1d-4a2a-bd25-e1e5068a26be-kube-api-access-6nkln\") pod \"maas-api-revoke-keys-e2e-ait-af87dbaf-qbvvf\" (UID: \"8df54091-af1d-4a2a-bd25-e1e5068a26be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af87dbaf-qbvvf" Oct 30 21:48:49.895592 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:49.895564 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af87dbaf-qbvvf" Oct 30 21:48:50.269963 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:50.269922 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af87dbaf-qbvvf"] Oct 30 21:48:50.274725 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:48:50.274695 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8df54091_af1d_4a2a_bd25_e1e5068a26be.slice/crio-59ee668b6bdaf6b9dff66388ffca99c632c3fcde223e37a7f0936c595c77845a WatchSource:0}: Error finding container 59ee668b6bdaf6b9dff66388ffca99c632c3fcde223e37a7f0936c595c77845a: Status 404 returned error can't find the container with id 59ee668b6bdaf6b9dff66388ffca99c632c3fcde223e37a7f0936c595c77845a Oct 30 21:48:51.011975 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:51.011945 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af87dbaf-qbvvf" event={"ID":"8df54091-af1d-4a2a-bd25-e1e5068a26be","Type":"ContainerStarted","Data":"69d6f90cad9a8ed49db17092576f2025cdab07d8156f36ab2858c7857d050935"} Oct 30 21:48:51.012381 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:51.011999 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af87dbaf-qbvvf" event={"ID":"8df54091-af1d-4a2a-bd25-e1e5068a26be","Type":"ContainerStarted","Data":"59ee668b6bdaf6b9dff66388ffca99c632c3fcde223e37a7f0936c595c77845a"} Oct 30 21:48:51.134282 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:51.134242 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af87dbaf-qbvvf" podStartSLOduration=2.134229254 podStartE2EDuration="2.134229254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:51.130564025 +0000 UTC m=+2106.685263802" watchObservedRunningTime="2026-10-30 21:48:51.134229254 +0000 UTC m=+2106.688929029" Oct 30 21:48:52.019874 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:52.019836 4560 generic.go:356] "Generic (PLEG): container finished" podID="8df54091-af1d-4a2a-bd25-e1e5068a26be" containerID="69d6f90cad9a8ed49db17092576f2025cdab07d8156f36ab2858c7857d050935" exitCode=0 Oct 30 21:48:52.020296 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:52.019914 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af87dbaf-qbvvf" event={"ID":"8df54091-af1d-4a2a-bd25-e1e5068a26be","Type":"ContainerDied","Data":"69d6f90cad9a8ed49db17092576f2025cdab07d8156f36ab2858c7857d050935"} Oct 30 21:48:53.056675 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:53.056638 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41ff2a70-2frst"] Oct 30 21:48:53.102588 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:53.102556 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41ff2a70-2frst"] Oct 30 21:48:53.191263 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:53.191236 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af87dbaf-qbvvf" Oct 30 21:48:53.289742 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:53.289666 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8df54091-af1d-4a2a-bd25-e1e5068a26be-kube-api-access-6nkln\" (UniqueName: \"kubernetes.io/projected/8df54091-af1d-4a2a-bd25-e1e5068a26be-kube-api-access-6nkln\") pod \"8df54091-af1d-4a2a-bd25-e1e5068a26be\" (UID: \"8df54091-af1d-4a2a-bd25-e1e5068a26be\") " Oct 30 21:48:53.289888 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:53.289772 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8df54091-af1d-4a2a-bd25-e1e5068a26be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8df54091-af1d-4a2a-bd25-e1e5068a26be-maas-api-openshift-ca\") pod \"8df54091-af1d-4a2a-bd25-e1e5068a26be\" (UID: \"8df54091-af1d-4a2a-bd25-e1e5068a26be\") " Oct 30 21:48:53.289888 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:53.289825 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8df54091-af1d-4a2a-bd25-e1e5068a26be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8df54091-af1d-4a2a-bd25-e1e5068a26be-maas-api-xks-ca\") pod \"8df54091-af1d-4a2a-bd25-e1e5068a26be\" (UID: \"8df54091-af1d-4a2a-bd25-e1e5068a26be\") " Oct 30 21:48:53.290133 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:53.290104 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8df54091-af1d-4a2a-bd25-e1e5068a26be-maas-api-openshift-ca" pod "8df54091-af1d-4a2a-bd25-e1e5068a26be" (UID: "8df54091-af1d-4a2a-bd25-e1e5068a26be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:53.290407 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:53.290383 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8df54091-af1d-4a2a-bd25-e1e5068a26be-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:53.291841 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:53.291819 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8df54091-af1d-4a2a-bd25-e1e5068a26be-kube-api-access-6nkln" pod "8df54091-af1d-4a2a-bd25-e1e5068a26be" (UID: "8df54091-af1d-4a2a-bd25-e1e5068a26be"). InnerVolumeSpecName "kube-api-access-6nkln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:53.291944 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:53.291897 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8df54091-af1d-4a2a-bd25-e1e5068a26be-maas-api-xks-ca" pod "8df54091-af1d-4a2a-bd25-e1e5068a26be" (UID: "8df54091-af1d-4a2a-bd25-e1e5068a26be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:53.392086 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:53.392023 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nkln\" (UniqueName: \"kubernetes.io/projected/8df54091-af1d-4a2a-bd25-e1e5068a26be-kube-api-access-6nkln\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:53.392086 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:53.392081 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8df54091-af1d-4a2a-bd25-e1e5068a26be-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:54.035855 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:54.035825 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af87dbaf-qbvvf" Oct 30 21:48:54.035855 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:54.035847 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af87dbaf-qbvvf" event={"ID":"8df54091-af1d-4a2a-bd25-e1e5068a26be","Type":"ContainerDied","Data":"59ee668b6bdaf6b9dff66388ffca99c632c3fcde223e37a7f0936c595c77845a"} Oct 30 21:48:54.036086 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:54.035895 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59ee668b6bdaf6b9dff66388ffca99c632c3fcde223e37a7f0936c595c77845a" Oct 30 21:48:54.577196 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:54.577164 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9361faed-bd30-4786-acb4-0dafcd9a87db" path="/var/lib/kubelet/pods/9361faed-bd30-4786-acb4-0dafcd9a87db/volumes" Oct 30 21:48:57.899416 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:57.899331 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e79f61de-4mxt7"] Oct 30 21:48:57.946089 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:57.946060 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e79f61de-4mxt7"] Oct 30 21:48:58.575874 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:48:58.575834 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1865ffda-3cf3-4f84-b111-d417ce2de3ec" path="/var/lib/kubelet/pods/1865ffda-3cf3-4f84-b111-d417ce2de3ec/volumes" Oct 30 21:49:05.253002 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:05.252969 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:49:05.826370 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:05.815222 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a84c1592-4swx2"] Oct 30 21:49:05.872712 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:05.872667 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a84c1592-4swx2"] Oct 30 21:49:06.649560 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:06.649072 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da33cb18-0fb2-47ee-a685-c2f1a72bd21e" path="/var/lib/kubelet/pods/da33cb18-0fb2-47ee-a685-c2f1a72bd21e/volumes" Oct 30 21:49:13.057369 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:13.057309 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8aa35089-qv44j"] Oct 30 21:49:13.103813 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:13.103783 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8aa35089-qv44j"] Oct 30 21:49:14.647362 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:14.647055 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="181d6638-2a2e-4718-bd93-a8df3cdb528b" path="/var/lib/kubelet/pods/181d6638-2a2e-4718-bd93-a8df3cdb528b/volumes" Oct 30 21:49:15.674930 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:15.674889 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-899a6bb1-84m6t"] Oct 30 21:49:15.721920 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:15.721886 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-899a6bb1-84m6t"] Oct 30 21:49:16.576631 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:16.576598 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4527511-5f50-4d62-bb3b-f6537457b147" path="/var/lib/kubelet/pods/e4527511-5f50-4d62-bb3b-f6537457b147/volumes" Oct 30 21:49:19.685095 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:49:19.685059 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:49:35.755762 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:35.755726 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af87dbaf-qbvvf"] Oct 30 21:49:35.801985 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:35.801951 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af87dbaf-qbvvf"] Oct 30 21:49:36.600714 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:36.600677 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8df54091-af1d-4a2a-bd25-e1e5068a26be" path="/var/lib/kubelet/pods/8df54091-af1d-4a2a-bd25-e1e5068a26be/volumes" Oct 30 21:49:38.076245 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:38.076210 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09bd6c1b-cp7jq"] Oct 30 21:49:38.123062 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:38.123034 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09bd6c1b-cp7jq"] Oct 30 21:49:38.576531 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:38.576493 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="633cda80-9a6a-4366-8dbd-8aa125655e2d" path="/var/lib/kubelet/pods/633cda80-9a6a-4366-8dbd-8aa125655e2d/volumes" Oct 30 21:49:41.486212 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:41.486172 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6fc664f-k7mpl"] Oct 30 21:49:41.486908 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:41.486886 4560 cpu_manager.go:414] "containerMap: removing container" podUID="8df54091-af1d-4a2a-bd25-e1e5068a26be" containerName="revoke-keys" Oct 30 21:49:41.486999 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:41.486908 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8df54091-af1d-4a2a-bd25-e1e5068a26be" containerName="revoke-keys" Oct 30 21:49:41.487089 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:41.487075 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="8df54091-af1d-4a2a-bd25-e1e5068a26be" containerName="revoke-keys" Oct 30 21:49:41.490751 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:41.490730 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6fc664f-k7mpl" Oct 30 21:49:41.497643 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:41.497556 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e85868aa-ad8d-4c14-8c39-0514ef644922-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6fc664f-k7mpl\" (UID: \"e85868aa-ad8d-4c14-8c39-0514ef644922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6fc664f-k7mpl" Oct 30 21:49:41.497715 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:41.497644 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssbgg\" (UniqueName: \"kubernetes.io/projected/e85868aa-ad8d-4c14-8c39-0514ef644922-kube-api-access-ssbgg\") pod \"maas-api-revoke-sub-b6fc664f-k7mpl\" (UID: \"e85868aa-ad8d-4c14-8c39-0514ef644922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6fc664f-k7mpl" Oct 30 21:49:41.497822 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:41.497805 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e85868aa-ad8d-4c14-8c39-0514ef644922-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6fc664f-k7mpl\" (UID: \"e85868aa-ad8d-4c14-8c39-0514ef644922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6fc664f-k7mpl" Oct 30 21:49:41.543250 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:41.543227 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:49:41.546219 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:41.546200 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6fc664f-k7mpl"] Oct 30 21:49:41.587308 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:41.587287 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:49:41.587461 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:41.587317 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:49:41.587461 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:41.587413 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:49:41.598606 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:41.598585 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e85868aa-ad8d-4c14-8c39-0514ef644922-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6fc664f-k7mpl\" (UID: \"e85868aa-ad8d-4c14-8c39-0514ef644922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6fc664f-k7mpl" Oct 30 21:49:41.598717 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:41.598643 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e85868aa-ad8d-4c14-8c39-0514ef644922-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6fc664f-k7mpl\" (UID: \"e85868aa-ad8d-4c14-8c39-0514ef644922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6fc664f-k7mpl" Oct 30 21:49:41.598777 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:41.598753 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssbgg\" (UniqueName: \"kubernetes.io/projected/e85868aa-ad8d-4c14-8c39-0514ef644922-kube-api-access-ssbgg\") pod \"maas-api-revoke-sub-b6fc664f-k7mpl\" (UID: \"e85868aa-ad8d-4c14-8c39-0514ef644922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6fc664f-k7mpl" Oct 30 21:49:41.599192 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:41.599173 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e85868aa-ad8d-4c14-8c39-0514ef644922-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6fc664f-k7mpl\" (UID: \"e85868aa-ad8d-4c14-8c39-0514ef644922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6fc664f-k7mpl" Oct 30 21:49:41.600892 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:41.600867 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e85868aa-ad8d-4c14-8c39-0514ef644922-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6fc664f-k7mpl\" (UID: \"e85868aa-ad8d-4c14-8c39-0514ef644922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6fc664f-k7mpl" Oct 30 21:49:41.656798 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:41.656776 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssbgg\" (UniqueName: \"kubernetes.io/projected/e85868aa-ad8d-4c14-8c39-0514ef644922-kube-api-access-ssbgg\") pod \"maas-api-revoke-sub-b6fc664f-k7mpl\" (UID: \"e85868aa-ad8d-4c14-8c39-0514ef644922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6fc664f-k7mpl" Oct 30 21:49:41.803624 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:41.803557 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6fc664f-k7mpl" Oct 30 21:49:42.193433 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:42.193335 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6fc664f-k7mpl"] Oct 30 21:49:42.197994 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:49:42.197969 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode85868aa_ad8d_4c14_8c39_0514ef644922.slice/crio-a24d80b0149d545bbe8b73da0d20fe7f79e959dcd02d43e8c73079c383249bdf WatchSource:0}: Error finding container a24d80b0149d545bbe8b73da0d20fe7f79e959dcd02d43e8c73079c383249bdf: Status 404 returned error can't find the container with id a24d80b0149d545bbe8b73da0d20fe7f79e959dcd02d43e8c73079c383249bdf Oct 30 21:49:42.525820 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:42.525785 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6fc664f-k7mpl" event={"ID":"e85868aa-ad8d-4c14-8c39-0514ef644922","Type":"ContainerStarted","Data":"a24d80b0149d545bbe8b73da0d20fe7f79e959dcd02d43e8c73079c383249bdf"} Oct 30 21:49:43.534449 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:43.534411 4560 generic.go:356] "Generic (PLEG): container finished" podID="e85868aa-ad8d-4c14-8c39-0514ef644922" containerID="3399edf97812039a0448a36c21245dc48cee4354065a5019738f5165b3e9ec41" exitCode=0 Oct 30 21:49:43.534871 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:43.534493 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6fc664f-k7mpl" event={"ID":"e85868aa-ad8d-4c14-8c39-0514ef644922","Type":"ContainerDied","Data":"3399edf97812039a0448a36c21245dc48cee4354065a5019738f5165b3e9ec41"} Oct 30 21:49:44.694697 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:44.694675 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6fc664f-k7mpl" Oct 30 21:49:44.737719 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:44.737697 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e85868aa-ad8d-4c14-8c39-0514ef644922-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e85868aa-ad8d-4c14-8c39-0514ef644922-maas-api-openshift-ca\") pod \"e85868aa-ad8d-4c14-8c39-0514ef644922\" (UID: \"e85868aa-ad8d-4c14-8c39-0514ef644922\") " Oct 30 21:49:44.737838 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:44.737821 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e85868aa-ad8d-4c14-8c39-0514ef644922-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e85868aa-ad8d-4c14-8c39-0514ef644922-maas-api-xks-ca\") pod \"e85868aa-ad8d-4c14-8c39-0514ef644922\" (UID: \"e85868aa-ad8d-4c14-8c39-0514ef644922\") " Oct 30 21:49:44.737878 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:44.737853 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e85868aa-ad8d-4c14-8c39-0514ef644922-kube-api-access-ssbgg\" (UniqueName: \"kubernetes.io/projected/e85868aa-ad8d-4c14-8c39-0514ef644922-kube-api-access-ssbgg\") pod \"e85868aa-ad8d-4c14-8c39-0514ef644922\" (UID: \"e85868aa-ad8d-4c14-8c39-0514ef644922\") " Oct 30 21:49:44.738009 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:44.737988 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e85868aa-ad8d-4c14-8c39-0514ef644922-maas-api-openshift-ca" pod "e85868aa-ad8d-4c14-8c39-0514ef644922" (UID: "e85868aa-ad8d-4c14-8c39-0514ef644922"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:44.738438 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:44.738415 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e85868aa-ad8d-4c14-8c39-0514ef644922-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:44.739815 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:44.739791 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e85868aa-ad8d-4c14-8c39-0514ef644922-maas-api-xks-ca" pod "e85868aa-ad8d-4c14-8c39-0514ef644922" (UID: "e85868aa-ad8d-4c14-8c39-0514ef644922"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:44.739906 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:44.739822 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e85868aa-ad8d-4c14-8c39-0514ef644922-kube-api-access-ssbgg" pod "e85868aa-ad8d-4c14-8c39-0514ef644922" (UID: "e85868aa-ad8d-4c14-8c39-0514ef644922"). InnerVolumeSpecName "kube-api-access-ssbgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:44.839901 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:44.839820 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e85868aa-ad8d-4c14-8c39-0514ef644922-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:44.839901 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:44.839847 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssbgg\" (UniqueName: \"kubernetes.io/projected/e85868aa-ad8d-4c14-8c39-0514ef644922-kube-api-access-ssbgg\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:45.550000 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:45.549972 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6fc664f-k7mpl" Oct 30 21:49:45.550000 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:45.549985 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6fc664f-k7mpl" event={"ID":"e85868aa-ad8d-4c14-8c39-0514ef644922","Type":"ContainerDied","Data":"a24d80b0149d545bbe8b73da0d20fe7f79e959dcd02d43e8c73079c383249bdf"} Oct 30 21:49:45.550290 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:45.550043 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a24d80b0149d545bbe8b73da0d20fe7f79e959dcd02d43e8c73079c383249bdf" Oct 30 21:49:46.125056 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:46.125021 4560 scope.go:122] "RemoveContainer" containerID="95a81b44ce200d767e61087e8b1cc2d6fda96886b8d55efb4389ee975acc3e42" Oct 30 21:49:49.054364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:49.054323 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc27d458-wwl72"] Oct 30 21:49:49.101016 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:49.100987 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc27d458-wwl72"] Oct 30 21:49:49.684733 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:49:49.684701 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:49:50.587775 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:50.587740 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="644973ad-c24e-4e73-bca3-d553a412da80" path="/var/lib/kubelet/pods/644973ad-c24e-4e73-bca3-d553a412da80/volumes" Oct 30 21:49:54.105358 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:54.074198 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-566bc88f-b98f9"] Oct 30 21:49:54.105940 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:54.105914 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-566bc88f-b98f9"] Oct 30 21:49:54.578410 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:49:54.578371 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7" path="/var/lib/kubelet/pods/b7d9d9a7-8bf9-438a-a9fe-5e9ab41150d7/volumes" Oct 30 21:50:03.050055 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:03.050018 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b967974-xbnj4"] Oct 30 21:50:03.096574 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:03.096541 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b967974-xbnj4"] Oct 30 21:50:04.390169 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:04.390125 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4"] Oct 30 21:50:04.392100 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:04.392064 4560 cpu_manager.go:414] "containerMap: removing container" podUID="e85868aa-ad8d-4c14-8c39-0514ef644922" containerName="revoke-keys" Oct 30 21:50:04.392282 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:04.392267 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e85868aa-ad8d-4c14-8c39-0514ef644922" containerName="revoke-keys" Oct 30 21:50:04.392575 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:04.392551 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="e85868aa-ad8d-4c14-8c39-0514ef644922" containerName="revoke-keys" Oct 30 21:50:04.414361 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:04.406122 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" Oct 30 21:50:04.510653 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:04.510619 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4"] Oct 30 21:50:04.553690 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:04.553656 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmgzw\" (UniqueName: \"kubernetes.io/projected/4c21f641-8dd8-4549-8748-94c9fdbd8ba2-kube-api-access-qmgzw\") pod \"maas-default-gateway-openshift-default-67bf986f99-gl8p4\" (UID: \"4c21f641-8dd8-4549-8748-94c9fdbd8ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" Oct 30 21:50:04.553877 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:04.553723 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c21f641-8dd8-4549-8748-94c9fdbd8ba2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-gl8p4\" (UID: \"4c21f641-8dd8-4549-8748-94c9fdbd8ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" Oct 30 21:50:04.553877 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:04.553838 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c21f641-8dd8-4549-8748-94c9fdbd8ba2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-gl8p4\" (UID: \"4c21f641-8dd8-4549-8748-94c9fdbd8ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" Oct 30 21:50:04.553952 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:04.553889 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c21f641-8dd8-4549-8748-94c9fdbd8ba2-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-gl8p4\" (UID: \"4c21f641-8dd8-4549-8748-94c9fdbd8ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" Oct 30 21:50:04.554017 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:04.554004 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c21f641-8dd8-4549-8748-94c9fdbd8ba2-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-gl8p4\" (UID: \"4c21f641-8dd8-4549-8748-94c9fdbd8ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" Oct 30 21:50:04.554057 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:04.554030 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c21f641-8dd8-4549-8748-94c9fdbd8ba2-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-gl8p4\" (UID: \"4c21f641-8dd8-4549-8748-94c9fdbd8ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" Oct 30 21:50:04.554111 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:04.554054 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c21f641-8dd8-4549-8748-94c9fdbd8ba2-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-gl8p4\" (UID: \"4c21f641-8dd8-4549-8748-94c9fdbd8ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" Oct 30 21:50:04.554169 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:04.554157 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c21f641-8dd8-4549-8748-94c9fdbd8ba2-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-gl8p4\" (UID: \"4c21f641-8dd8-4549-8748-94c9fdbd8ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" Oct 30 21:50:04.554221 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:04.554179 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c21f641-8dd8-4549-8748-94c9fdbd8ba2-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-gl8p4\" (UID: \"4c21f641-8dd8-4549-8748-94c9fdbd8ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" Oct 30 21:50:04.576536 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:04.576505 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b2a8f19-1112-42d5-b6cb-f6625301f86e" path="/var/lib/kubelet/pods/4b2a8f19-1112-42d5-b6cb-f6625301f86e/volumes" Oct 30 21:50:04.655698 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:04.655609 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmgzw\" (UniqueName: \"kubernetes.io/projected/4c21f641-8dd8-4549-8748-94c9fdbd8ba2-kube-api-access-qmgzw\") pod \"maas-default-gateway-openshift-default-67bf986f99-gl8p4\" (UID: \"4c21f641-8dd8-4549-8748-94c9fdbd8ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" Oct 30 21:50:04.655883 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:04.655822 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c21f641-8dd8-4549-8748-94c9fdbd8ba2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-gl8p4\" (UID: \"4c21f641-8dd8-4549-8748-94c9fdbd8ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" Oct 30 21:50:04.655883 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:04.655869 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c21f641-8dd8-4549-8748-94c9fdbd8ba2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-gl8p4\" (UID: \"4c21f641-8dd8-4549-8748-94c9fdbd8ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" Oct 30 21:50:04.656011 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:04.655895 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c21f641-8dd8-4549-8748-94c9fdbd8ba2-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-gl8p4\" (UID: \"4c21f641-8dd8-4549-8748-94c9fdbd8ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" Oct 30 21:50:04.656077 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:04.656022 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c21f641-8dd8-4549-8748-94c9fdbd8ba2-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-gl8p4\" (UID: \"4c21f641-8dd8-4549-8748-94c9fdbd8ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" Oct 30 21:50:04.656077 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:04.656058 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c21f641-8dd8-4549-8748-94c9fdbd8ba2-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-gl8p4\" (UID: \"4c21f641-8dd8-4549-8748-94c9fdbd8ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" Oct 30 21:50:04.656196 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:04.656102 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c21f641-8dd8-4549-8748-94c9fdbd8ba2-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-gl8p4\" (UID: \"4c21f641-8dd8-4549-8748-94c9fdbd8ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" Oct 30 21:50:04.656196 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:04.656159 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c21f641-8dd8-4549-8748-94c9fdbd8ba2-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-gl8p4\" (UID: \"4c21f641-8dd8-4549-8748-94c9fdbd8ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" Oct 30 21:50:04.656196 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:04.656188 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c21f641-8dd8-4549-8748-94c9fdbd8ba2-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-gl8p4\" (UID: \"4c21f641-8dd8-4549-8748-94c9fdbd8ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" Oct 30 21:50:04.656722 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:04.656602 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c21f641-8dd8-4549-8748-94c9fdbd8ba2-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-gl8p4\" (UID: \"4c21f641-8dd8-4549-8748-94c9fdbd8ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" Oct 30 21:50:04.656722 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:04.656612 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c21f641-8dd8-4549-8748-94c9fdbd8ba2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-gl8p4\" (UID: \"4c21f641-8dd8-4549-8748-94c9fdbd8ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" Oct 30 21:50:04.656722 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:04.656631 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c21f641-8dd8-4549-8748-94c9fdbd8ba2-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-gl8p4\" (UID: \"4c21f641-8dd8-4549-8748-94c9fdbd8ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" Oct 30 21:50:04.656722 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:04.656660 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c21f641-8dd8-4549-8748-94c9fdbd8ba2-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-gl8p4\" (UID: \"4c21f641-8dd8-4549-8748-94c9fdbd8ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" Oct 30 21:50:04.656946 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:04.656845 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c21f641-8dd8-4549-8748-94c9fdbd8ba2-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-gl8p4\" (UID: \"4c21f641-8dd8-4549-8748-94c9fdbd8ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" Oct 30 21:50:04.658136 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:04.658119 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c21f641-8dd8-4549-8748-94c9fdbd8ba2-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-gl8p4\" (UID: \"4c21f641-8dd8-4549-8748-94c9fdbd8ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" Oct 30 21:50:04.658751 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:04.658734 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c21f641-8dd8-4549-8748-94c9fdbd8ba2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-gl8p4\" (UID: \"4c21f641-8dd8-4549-8748-94c9fdbd8ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" Oct 30 21:50:04.714697 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:04.714666 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmgzw\" (UniqueName: \"kubernetes.io/projected/4c21f641-8dd8-4549-8748-94c9fdbd8ba2-kube-api-access-qmgzw\") pod \"maas-default-gateway-openshift-default-67bf986f99-gl8p4\" (UID: \"4c21f641-8dd8-4549-8748-94c9fdbd8ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" Oct 30 21:50:04.723543 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:04.723519 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c21f641-8dd8-4549-8748-94c9fdbd8ba2-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-gl8p4\" (UID: \"4c21f641-8dd8-4549-8748-94c9fdbd8ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" Oct 30 21:50:04.739652 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:04.739629 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" Oct 30 21:50:05.177787 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:05.177737 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4"] Oct 30 21:50:05.182247 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:50:05.182219 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c21f641_8dd8_4549_8748_94c9fdbd8ba2.slice/crio-e49bd07646eaf2ad69fd658d3866279768361a4d5681749d627d902b9eca8e73 WatchSource:0}: Error finding container e49bd07646eaf2ad69fd658d3866279768361a4d5681749d627d902b9eca8e73: Status 404 returned error can't find the container with id e49bd07646eaf2ad69fd658d3866279768361a4d5681749d627d902b9eca8e73 Oct 30 21:50:05.184660 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:05.184627 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:50:05.184737 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:05.184699 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:50:05.184777 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:05.184748 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:50:05.694674 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:05.694637 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" event={"ID":"4c21f641-8dd8-4549-8748-94c9fdbd8ba2","Type":"ContainerStarted","Data":"f9ca415eda61c1260e91fa46178979f580e82766b8b10eb045835b62876e5640"} Oct 30 21:50:05.695057 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:05.694685 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" event={"ID":"4c21f641-8dd8-4549-8748-94c9fdbd8ba2","Type":"ContainerStarted","Data":"e49bd07646eaf2ad69fd658d3866279768361a4d5681749d627d902b9eca8e73"} Oct 30 21:50:05.740191 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:05.740164 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" Oct 30 21:50:05.812967 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:05.812911 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" podStartSLOduration=1.812897048 podStartE2EDuration="1.812897048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:05.810582277 +0000 UTC m=+2181.365282050" watchObservedRunningTime="2026-10-30 21:50:05.812897048 +0000 UTC m=+2181.367596822" Oct 30 21:50:06.753355 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:06.753297 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-gl8p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 21:50:06.753866 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:06.753368 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" podUID="4c21f641-8dd8-4549-8748-94c9fdbd8ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 21:50:07.749692 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:07.749658 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-gl8p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 21:50:07.749976 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:07.749948 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" podUID="4c21f641-8dd8-4549-8748-94c9fdbd8ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 21:50:08.746364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:08.742884 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-gl8p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 21:50:08.746364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:08.742952 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" podUID="4c21f641-8dd8-4549-8748-94c9fdbd8ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 21:50:09.478883 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:09.478848 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:50:09.750529 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:09.750438 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-gl8p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 21:50:09.751483 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:09.751443 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" podUID="4c21f641-8dd8-4549-8748-94c9fdbd8ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 21:50:10.741270 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:10.741233 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-gl8p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 21:50:10.741654 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:10.741623 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" podUID="4c21f641-8dd8-4549-8748-94c9fdbd8ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 21:50:11.119172 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:11.119104 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-637696f1-pvfq6"] Oct 30 21:50:11.122764 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:11.122741 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-637696f1-pvfq6"] Oct 30 21:50:11.740602 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:11.740563 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-gl8p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 21:50:11.740811 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:11.740631 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" podUID="4c21f641-8dd8-4549-8748-94c9fdbd8ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 21:50:12.581104 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:12.581057 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c" path="/var/lib/kubelet/pods/cf67bd40-7ae8-45c7-97f3-3e89f2ddef5c/volumes" Oct 30 21:50:12.740355 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:12.740304 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-gl8p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 21:50:12.740521 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:12.740391 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" podUID="4c21f641-8dd8-4549-8748-94c9fdbd8ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 21:50:14.066130 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:14.066102 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a01ea9f-2pcg5"] Oct 30 21:50:14.115456 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:14.115421 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a01ea9f-2pcg5"] Oct 30 21:50:14.577939 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:14.577908 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2efef570-20ac-4b9e-9d4e-f78bc752c0a1" path="/var/lib/kubelet/pods/2efef570-20ac-4b9e-9d4e-f78bc752c0a1/volumes" Oct 30 21:50:14.740943 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:14.740906 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-gl8p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:50:14.741128 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:14.740967 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" podUID="4c21f641-8dd8-4549-8748-94c9fdbd8ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:50:15.782509 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:15.782466 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-gl8p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:50:15.783025 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:15.782533 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" podUID="4c21f641-8dd8-4549-8748-94c9fdbd8ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:50:16.130806 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:16.130725 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" Oct 30 21:50:16.131156 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:16.131136 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" Oct 30 21:50:16.131801 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:16.131782 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" Oct 30 21:50:19.048560 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:19.048527 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78afec61-7vcnz"] Oct 30 21:50:19.095501 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:19.095471 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78afec61-7vcnz"] Oct 30 21:50:19.690711 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:50:19.690678 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:50:20.577541 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:20.577512 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abbc55ac-7b18-4926-a0ba-7217a13bdc41" path="/var/lib/kubelet/pods/abbc55ac-7b18-4926-a0ba-7217a13bdc41/volumes" Oct 30 21:50:24.054221 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:24.054188 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75ee6b3c-fxrpq"] Oct 30 21:50:24.100926 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:24.100897 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75ee6b3c-fxrpq"] Oct 30 21:50:24.577404 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:24.577375 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9efba11d-653c-47de-9eaf-dc250f12d18c" path="/var/lib/kubelet/pods/9efba11d-653c-47de-9eaf-dc250f12d18c/volumes" Oct 30 21:50:35.107507 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:35.107427 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d0e8d90-8g7xn"] Oct 30 21:50:35.153794 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:35.153757 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d0e8d90-8g7xn"] Oct 30 21:50:36.578202 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:36.578166 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="567f946c-fb5c-4798-b11c-bae6723f3d89" path="/var/lib/kubelet/pods/567f946c-fb5c-4798-b11c-bae6723f3d89/volumes" Oct 30 21:50:37.048747 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:37.048714 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70cf0de4-b9pvm"] Oct 30 21:50:37.095631 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:37.095598 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70cf0de4-b9pvm"] Oct 30 21:50:38.053988 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:38.053956 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a878660f-q4pkx"] Oct 30 21:50:38.100392 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:38.100366 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a878660f-q4pkx"] Oct 30 21:50:38.578243 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:38.578206 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aaa2a11-2803-4d9d-9bf2-75175375cb8f" path="/var/lib/kubelet/pods/7aaa2a11-2803-4d9d-9bf2-75175375cb8f/volumes" Oct 30 21:50:38.578788 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:38.578773 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ace52bf5-0ea8-4d12-94b9-e234c93f8720" path="/var/lib/kubelet/pods/ace52bf5-0ea8-4d12-94b9-e234c93f8720/volumes" Oct 30 21:50:40.084208 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:40.084172 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ea0c294-8l2dw"] Oct 30 21:50:40.119452 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:40.119418 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40a39e54-dpfcc"] Oct 30 21:50:40.132413 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:40.132381 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ea0c294-8l2dw"] Oct 30 21:50:40.142880 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:40.142829 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40a39e54-dpfcc"] Oct 30 21:50:40.577762 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:40.577726 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="223b6566-cc1e-49f4-866f-d8e09df28cd9" path="/var/lib/kubelet/pods/223b6566-cc1e-49f4-866f-d8e09df28cd9/volumes" Oct 30 21:50:40.578268 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:40.578246 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d196a58-9be7-41e1-93cc-2348c2f7c30c" path="/var/lib/kubelet/pods/6d196a58-9be7-41e1-93cc-2348c2f7c30c/volumes" Oct 30 21:50:46.234101 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:46.234071 4560 scope.go:122] "RemoveContainer" containerID="344ff5e284e30ec0d9d96e9248ec3b175b529167a9650694f03b863e161ebe87" Oct 30 21:50:46.250011 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:46.249981 4560 scope.go:122] "RemoveContainer" containerID="5416453a2ce236a819cf445b54b509acf10bd74da949ddbd57ddc9a9d07b7bed" Oct 30 21:50:46.265532 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:46.265507 4560 scope.go:122] "RemoveContainer" containerID="6b2f8fad10de1dea3c3a91319938717e31fcda5e3ec094adcff1be1efe18d7c4" Oct 30 21:50:46.922769 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:46.922736 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-364e1518-nc9sd"] Oct 30 21:50:46.928497 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:46.928476 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-364e1518-nc9sd" Oct 30 21:50:47.025211 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:47.025178 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:50:47.025401 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:47.025220 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:50:47.025401 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:47.025265 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:50:47.025401 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:47.025265 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:50:47.028793 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:47.028773 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-364e1518-nc9sd"] Oct 30 21:50:47.053179 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:47.053155 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f9a82e0-3d49-40ef-a1fc-405f179cbf77-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-364e1518-nc9sd\" (UID: \"2f9a82e0-3d49-40ef-a1fc-405f179cbf77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-364e1518-nc9sd" Oct 30 21:50:47.053293 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:47.053220 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjhz7\" (UniqueName: \"kubernetes.io/projected/2f9a82e0-3d49-40ef-a1fc-405f179cbf77-kube-api-access-jjhz7\") pod \"maas-api-revoke-keys-e2e-mt-364e1518-nc9sd\" (UID: \"2f9a82e0-3d49-40ef-a1fc-405f179cbf77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-364e1518-nc9sd" Oct 30 21:50:47.053369 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:47.053306 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f9a82e0-3d49-40ef-a1fc-405f179cbf77-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-364e1518-nc9sd\" (UID: \"2f9a82e0-3d49-40ef-a1fc-405f179cbf77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-364e1518-nc9sd" Oct 30 21:50:47.154452 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:47.154422 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f9a82e0-3d49-40ef-a1fc-405f179cbf77-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-364e1518-nc9sd\" (UID: \"2f9a82e0-3d49-40ef-a1fc-405f179cbf77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-364e1518-nc9sd" Oct 30 21:50:47.154599 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:47.154500 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjhz7\" (UniqueName: \"kubernetes.io/projected/2f9a82e0-3d49-40ef-a1fc-405f179cbf77-kube-api-access-jjhz7\") pod \"maas-api-revoke-keys-e2e-mt-364e1518-nc9sd\" (UID: \"2f9a82e0-3d49-40ef-a1fc-405f179cbf77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-364e1518-nc9sd" Oct 30 21:50:47.154599 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:47.154530 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f9a82e0-3d49-40ef-a1fc-405f179cbf77-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-364e1518-nc9sd\" (UID: \"2f9a82e0-3d49-40ef-a1fc-405f179cbf77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-364e1518-nc9sd" Oct 30 21:50:47.155152 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:47.155134 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f9a82e0-3d49-40ef-a1fc-405f179cbf77-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-364e1518-nc9sd\" (UID: \"2f9a82e0-3d49-40ef-a1fc-405f179cbf77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-364e1518-nc9sd" Oct 30 21:50:47.156642 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:47.156615 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f9a82e0-3d49-40ef-a1fc-405f179cbf77-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-364e1518-nc9sd\" (UID: \"2f9a82e0-3d49-40ef-a1fc-405f179cbf77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-364e1518-nc9sd" Oct 30 21:50:47.212394 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:47.212319 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjhz7\" (UniqueName: \"kubernetes.io/projected/2f9a82e0-3d49-40ef-a1fc-405f179cbf77-kube-api-access-jjhz7\") pod \"maas-api-revoke-keys-e2e-mt-364e1518-nc9sd\" (UID: \"2f9a82e0-3d49-40ef-a1fc-405f179cbf77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-364e1518-nc9sd" Oct 30 21:50:47.241932 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:47.241907 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-364e1518-nc9sd" Oct 30 21:50:47.614877 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:47.614842 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-364e1518-nc9sd"] Oct 30 21:50:47.619989 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:50:47.619954 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f9a82e0_3d49_40ef_a1fc_405f179cbf77.slice/crio-96bebf31231df0c2e152ccc1bf2a46f8c8ac7865776104de5007f1436be58107 WatchSource:0}: Error finding container 96bebf31231df0c2e152ccc1bf2a46f8c8ac7865776104de5007f1436be58107: Status 404 returned error can't find the container with id 96bebf31231df0c2e152ccc1bf2a46f8c8ac7865776104de5007f1436be58107 Oct 30 21:50:48.054476 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:48.054435 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-364e1518-nc9sd" event={"ID":"2f9a82e0-3d49-40ef-a1fc-405f179cbf77","Type":"ContainerStarted","Data":"956e10023d9401776ea08ed95e448b1fb5cfdd97098e6c7bd4ee7ba0fefec90f"} Oct 30 21:50:48.054621 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:48.054492 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-364e1518-nc9sd" event={"ID":"2f9a82e0-3d49-40ef-a1fc-405f179cbf77","Type":"ContainerStarted","Data":"96bebf31231df0c2e152ccc1bf2a46f8c8ac7865776104de5007f1436be58107"} Oct 30 21:50:48.165915 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:48.165868 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-364e1518-nc9sd" podStartSLOduration=2.165854242 podStartE2EDuration="2.165854242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:48.164022155 +0000 UTC m=+2223.718721931" watchObservedRunningTime="2026-10-30 21:50:48.165854242 +0000 UTC m=+2223.720554034" Oct 30 21:50:49.062847 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:49.062812 4560 generic.go:356] "Generic (PLEG): container finished" podID="2f9a82e0-3d49-40ef-a1fc-405f179cbf77" containerID="956e10023d9401776ea08ed95e448b1fb5cfdd97098e6c7bd4ee7ba0fefec90f" exitCode=0 Oct 30 21:50:49.063214 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:49.062894 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-364e1518-nc9sd" event={"ID":"2f9a82e0-3d49-40ef-a1fc-405f179cbf77","Type":"ContainerDied","Data":"956e10023d9401776ea08ed95e448b1fb5cfdd97098e6c7bd4ee7ba0fefec90f"} Oct 30 21:50:49.683983 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:50:49.683951 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:50:50.056444 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:50.056407 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-910e9fa6-ghrtx"] Oct 30 21:50:50.103049 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:50.103020 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-910e9fa6-ghrtx"] Oct 30 21:50:50.210997 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:50.210975 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-364e1518-nc9sd" Oct 30 21:50:50.297532 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:50.297506 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f9a82e0-3d49-40ef-a1fc-405f179cbf77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f9a82e0-3d49-40ef-a1fc-405f179cbf77-maas-api-xks-ca\") pod \"2f9a82e0-3d49-40ef-a1fc-405f179cbf77\" (UID: \"2f9a82e0-3d49-40ef-a1fc-405f179cbf77\") " Oct 30 21:50:50.297752 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:50.297736 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f9a82e0-3d49-40ef-a1fc-405f179cbf77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f9a82e0-3d49-40ef-a1fc-405f179cbf77-maas-api-openshift-ca\") pod \"2f9a82e0-3d49-40ef-a1fc-405f179cbf77\" (UID: \"2f9a82e0-3d49-40ef-a1fc-405f179cbf77\") " Oct 30 21:50:50.297825 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:50.297784 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f9a82e0-3d49-40ef-a1fc-405f179cbf77-kube-api-access-jjhz7\" (UniqueName: \"kubernetes.io/projected/2f9a82e0-3d49-40ef-a1fc-405f179cbf77-kube-api-access-jjhz7\") pod \"2f9a82e0-3d49-40ef-a1fc-405f179cbf77\" (UID: \"2f9a82e0-3d49-40ef-a1fc-405f179cbf77\") " Oct 30 21:50:50.298065 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:50.298042 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f9a82e0-3d49-40ef-a1fc-405f179cbf77-maas-api-openshift-ca" pod "2f9a82e0-3d49-40ef-a1fc-405f179cbf77" (UID: "2f9a82e0-3d49-40ef-a1fc-405f179cbf77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:50.298309 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:50.298283 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f9a82e0-3d49-40ef-a1fc-405f179cbf77-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:50.299545 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:50.299522 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f9a82e0-3d49-40ef-a1fc-405f179cbf77-maas-api-xks-ca" pod "2f9a82e0-3d49-40ef-a1fc-405f179cbf77" (UID: "2f9a82e0-3d49-40ef-a1fc-405f179cbf77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:50.299691 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:50.299677 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f9a82e0-3d49-40ef-a1fc-405f179cbf77-kube-api-access-jjhz7" pod "2f9a82e0-3d49-40ef-a1fc-405f179cbf77" (UID: "2f9a82e0-3d49-40ef-a1fc-405f179cbf77"). InnerVolumeSpecName "kube-api-access-jjhz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:50.399244 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:50.399216 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjhz7\" (UniqueName: \"kubernetes.io/projected/2f9a82e0-3d49-40ef-a1fc-405f179cbf77-kube-api-access-jjhz7\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:50.399244 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:50.399240 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f9a82e0-3d49-40ef-a1fc-405f179cbf77-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:50.576573 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:50.576497 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e109d452-56a4-47f8-bbd6-cea3421ad8c7" path="/var/lib/kubelet/pods/e109d452-56a4-47f8-bbd6-cea3421ad8c7/volumes" Oct 30 21:50:51.077261 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:51.077229 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-364e1518-nc9sd" Oct 30 21:50:51.077432 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:51.077278 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-364e1518-nc9sd" event={"ID":"2f9a82e0-3d49-40ef-a1fc-405f179cbf77","Type":"ContainerDied","Data":"96bebf31231df0c2e152ccc1bf2a46f8c8ac7865776104de5007f1436be58107"} Oct 30 21:50:51.077432 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:51.077326 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96bebf31231df0c2e152ccc1bf2a46f8c8ac7865776104de5007f1436be58107" Oct 30 21:50:51.113538 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:51.113507 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx"] Oct 30 21:50:51.113961 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:51.113736 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" podUID="2bb9a4d9-6480-470b-967f-96b99f9ff606" containerName="istio-proxy" containerID="cri-o://95ec0a84a1a56bce468788fb97dad5fe4d70f7ff4d155f9f804e617f90b63930" gracePeriod=30 Oct 30 21:50:51.318915 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:51.318883 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fbdbd87a-mmptb"] Oct 30 21:50:51.319427 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:51.319413 4560 cpu_manager.go:414] "containerMap: removing container" podUID="2f9a82e0-3d49-40ef-a1fc-405f179cbf77" containerName="revoke-keys" Oct 30 21:50:51.319479 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:51.319428 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f9a82e0-3d49-40ef-a1fc-405f179cbf77" containerName="revoke-keys" Oct 30 21:50:51.319547 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:51.319538 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f9a82e0-3d49-40ef-a1fc-405f179cbf77" containerName="revoke-keys" Oct 30 21:50:51.323170 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:51.323150 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fbdbd87a-mmptb" Oct 30 21:50:51.377691 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:51.377664 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:50:51.377884 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:51.377707 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:50:51.377884 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:51.377837 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:50:51.378050 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:51.377865 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:50:51.379271 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:51.379254 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fbdbd87a-mmptb"] Oct 30 21:50:51.411068 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:51.411036 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e92ada-e6f3-4b6a-b82f-728481f30aff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fbdbd87a-mmptb\" (UID: \"60e92ada-e6f3-4b6a-b82f-728481f30aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fbdbd87a-mmptb" Oct 30 21:50:51.411216 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:51.411154 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qtlb\" (UniqueName: \"kubernetes.io/projected/60e92ada-e6f3-4b6a-b82f-728481f30aff-kube-api-access-2qtlb\") pod \"maas-api-revoke-keys-e2e-migrate-fbdbd87a-mmptb\" (UID: \"60e92ada-e6f3-4b6a-b82f-728481f30aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fbdbd87a-mmptb" Oct 30 21:50:51.411293 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:51.411231 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e92ada-e6f3-4b6a-b82f-728481f30aff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fbdbd87a-mmptb\" (UID: \"60e92ada-e6f3-4b6a-b82f-728481f30aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fbdbd87a-mmptb" Oct 30 21:50:51.516284 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:51.516248 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e92ada-e6f3-4b6a-b82f-728481f30aff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fbdbd87a-mmptb\" (UID: \"60e92ada-e6f3-4b6a-b82f-728481f30aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fbdbd87a-mmptb" Oct 30 21:50:51.516980 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:51.516480 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e92ada-e6f3-4b6a-b82f-728481f30aff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fbdbd87a-mmptb\" (UID: \"60e92ada-e6f3-4b6a-b82f-728481f30aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fbdbd87a-mmptb" Oct 30 21:50:51.516980 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:51.516584 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qtlb\" (UniqueName: \"kubernetes.io/projected/60e92ada-e6f3-4b6a-b82f-728481f30aff-kube-api-access-2qtlb\") pod \"maas-api-revoke-keys-e2e-migrate-fbdbd87a-mmptb\" (UID: \"60e92ada-e6f3-4b6a-b82f-728481f30aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fbdbd87a-mmptb" Oct 30 21:50:51.529360 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:51.517799 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e92ada-e6f3-4b6a-b82f-728481f30aff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fbdbd87a-mmptb\" (UID: \"60e92ada-e6f3-4b6a-b82f-728481f30aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fbdbd87a-mmptb" Oct 30 21:50:51.529360 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:51.525487 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e92ada-e6f3-4b6a-b82f-728481f30aff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fbdbd87a-mmptb\" (UID: \"60e92ada-e6f3-4b6a-b82f-728481f30aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fbdbd87a-mmptb" Oct 30 21:50:51.575243 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:51.575209 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qtlb\" (UniqueName: \"kubernetes.io/projected/60e92ada-e6f3-4b6a-b82f-728481f30aff-kube-api-access-2qtlb\") pod \"maas-api-revoke-keys-e2e-migrate-fbdbd87a-mmptb\" (UID: \"60e92ada-e6f3-4b6a-b82f-728481f30aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fbdbd87a-mmptb" Oct 30 21:50:51.641450 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:51.641382 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fbdbd87a-mmptb" Oct 30 21:50:52.031084 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:52.031051 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fbdbd87a-mmptb"] Oct 30 21:50:52.040278 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:50:52.040252 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e92ada_e6f3_4b6a_b82f_728481f30aff.slice/crio-a7084ccd261a10c6fb4727b9ba4d4cb6ae546a1ec3c7a83fbce6bbe9f73ab4f4 WatchSource:0}: Error finding container a7084ccd261a10c6fb4727b9ba4d4cb6ae546a1ec3c7a83fbce6bbe9f73ab4f4: Status 404 returned error can't find the container with id a7084ccd261a10c6fb4727b9ba4d4cb6ae546a1ec3c7a83fbce6bbe9f73ab4f4 Oct 30 21:50:52.085070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:52.085044 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fbdbd87a-mmptb" event={"ID":"60e92ada-e6f3-4b6a-b82f-728481f30aff","Type":"ContainerStarted","Data":"a7084ccd261a10c6fb4727b9ba4d4cb6ae546a1ec3c7a83fbce6bbe9f73ab4f4"} Oct 30 21:50:53.094227 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:53.094186 4560 generic.go:356] "Generic (PLEG): container finished" podID="60e92ada-e6f3-4b6a-b82f-728481f30aff" containerID="e78ed765ce8e17926acb42d69efaaeceee1012b83830837477751f62b1786c91" exitCode=0 Oct 30 21:50:53.094597 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:53.094244 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fbdbd87a-mmptb" event={"ID":"60e92ada-e6f3-4b6a-b82f-728481f30aff","Type":"ContainerDied","Data":"e78ed765ce8e17926acb42d69efaaeceee1012b83830837477751f62b1786c91"} Oct 30 21:50:54.048724 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:54.048690 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf6e2-w9zhs"] Oct 30 21:50:54.095529 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:54.095494 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf6e2-w9zhs"] Oct 30 21:50:54.266030 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:54.266009 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fbdbd87a-mmptb" Oct 30 21:50:54.350421 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:54.350367 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60e92ada-e6f3-4b6a-b82f-728481f30aff-kube-api-access-2qtlb\" (UniqueName: \"kubernetes.io/projected/60e92ada-e6f3-4b6a-b82f-728481f30aff-kube-api-access-2qtlb\") pod \"60e92ada-e6f3-4b6a-b82f-728481f30aff\" (UID: \"60e92ada-e6f3-4b6a-b82f-728481f30aff\") " Oct 30 21:50:54.350421 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:54.350415 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60e92ada-e6f3-4b6a-b82f-728481f30aff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e92ada-e6f3-4b6a-b82f-728481f30aff-maas-api-openshift-ca\") pod \"60e92ada-e6f3-4b6a-b82f-728481f30aff\" (UID: \"60e92ada-e6f3-4b6a-b82f-728481f30aff\") " Oct 30 21:50:54.350608 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:54.350544 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60e92ada-e6f3-4b6a-b82f-728481f30aff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e92ada-e6f3-4b6a-b82f-728481f30aff-maas-api-xks-ca\") pod \"60e92ada-e6f3-4b6a-b82f-728481f30aff\" (UID: \"60e92ada-e6f3-4b6a-b82f-728481f30aff\") " Oct 30 21:50:54.350789 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:54.350764 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60e92ada-e6f3-4b6a-b82f-728481f30aff-maas-api-openshift-ca" pod "60e92ada-e6f3-4b6a-b82f-728481f30aff" (UID: "60e92ada-e6f3-4b6a-b82f-728481f30aff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:54.351093 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:54.351076 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e92ada-e6f3-4b6a-b82f-728481f30aff-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:54.352301 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:54.352284 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e92ada-e6f3-4b6a-b82f-728481f30aff-kube-api-access-2qtlb" pod "60e92ada-e6f3-4b6a-b82f-728481f30aff" (UID: "60e92ada-e6f3-4b6a-b82f-728481f30aff"). InnerVolumeSpecName "kube-api-access-2qtlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:54.352498 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:54.352481 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60e92ada-e6f3-4b6a-b82f-728481f30aff-maas-api-xks-ca" pod "60e92ada-e6f3-4b6a-b82f-728481f30aff" (UID: "60e92ada-e6f3-4b6a-b82f-728481f30aff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:54.452834 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:54.452805 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e92ada-e6f3-4b6a-b82f-728481f30aff-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:54.452834 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:54.452830 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qtlb\" (UniqueName: \"kubernetes.io/projected/60e92ada-e6f3-4b6a-b82f-728481f30aff-kube-api-access-2qtlb\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:54.576638 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:54.576608 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d69cebfb-2af0-4064-b06e-9a4e3a09d43b" path="/var/lib/kubelet/pods/d69cebfb-2af0-4064-b06e-9a4e3a09d43b/volumes" Oct 30 21:50:55.109472 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:55.109438 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fbdbd87a-mmptb" Oct 30 21:50:55.109862 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:55.109479 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fbdbd87a-mmptb" event={"ID":"60e92ada-e6f3-4b6a-b82f-728481f30aff","Type":"ContainerDied","Data":"a7084ccd261a10c6fb4727b9ba4d4cb6ae546a1ec3c7a83fbce6bbe9f73ab4f4"} Oct 30 21:50:55.109862 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:55.109521 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7084ccd261a10c6fb4727b9ba4d4cb6ae546a1ec3c7a83fbce6bbe9f73ab4f4" Oct 30 21:50:56.053074 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:56.053040 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4022d31-wlsbm"] Oct 30 21:50:56.101365 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:56.101315 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87abedcf-cshsf"] Oct 30 21:50:56.104639 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:56.104613 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4022d31-wlsbm"] Oct 30 21:50:56.107278 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:56.107257 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87abedcf-cshsf"] Oct 30 21:50:56.485697 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:56.485673 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" Oct 30 21:50:56.576204 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:56.576178 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43c13f33-cdc2-4f36-85c4-956b1955a361" path="/var/lib/kubelet/pods/43c13f33-cdc2-4f36-85c4-956b1955a361/volumes" Oct 30 21:50:56.576615 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:56.576601 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e03c1ba-6336-4b4e-aa29-f9700babe1fa" path="/var/lib/kubelet/pods/5e03c1ba-6336-4b4e-aa29-f9700babe1fa/volumes" Oct 30 21:50:56.598751 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:56.598731 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bb9a4d9-6480-470b-967f-96b99f9ff606-kube-api-access-s8fdk\" (UniqueName: \"kubernetes.io/projected/2bb9a4d9-6480-470b-967f-96b99f9ff606-kube-api-access-s8fdk\") pod \"2bb9a4d9-6480-470b-967f-96b99f9ff606\" (UID: \"2bb9a4d9-6480-470b-967f-96b99f9ff606\") " Oct 30 21:50:56.598863 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:56.598771 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb9a4d9-6480-470b-967f-96b99f9ff606-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb9a4d9-6480-470b-967f-96b99f9ff606-credential-socket\") pod \"2bb9a4d9-6480-470b-967f-96b99f9ff606\" (UID: \"2bb9a4d9-6480-470b-967f-96b99f9ff606\") " Oct 30 21:50:56.598863 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:56.598823 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb9a4d9-6480-470b-967f-96b99f9ff606-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb9a4d9-6480-470b-967f-96b99f9ff606-istio-data\") pod \"2bb9a4d9-6480-470b-967f-96b99f9ff606\" (UID: \"2bb9a4d9-6480-470b-967f-96b99f9ff606\") " Oct 30 21:50:56.598973 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:56.598858 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb9a4d9-6480-470b-967f-96b99f9ff606-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb9a4d9-6480-470b-967f-96b99f9ff606-workload-certs\") pod \"2bb9a4d9-6480-470b-967f-96b99f9ff606\" (UID: \"2bb9a4d9-6480-470b-967f-96b99f9ff606\") " Oct 30 21:50:56.598973 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:56.598892 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bb9a4d9-6480-470b-967f-96b99f9ff606-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb9a4d9-6480-470b-967f-96b99f9ff606-istiod-ca-cert\") pod \"2bb9a4d9-6480-470b-967f-96b99f9ff606\" (UID: \"2bb9a4d9-6480-470b-967f-96b99f9ff606\") " Oct 30 21:50:56.598973 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:56.598946 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2bb9a4d9-6480-470b-967f-96b99f9ff606-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb9a4d9-6480-470b-967f-96b99f9ff606-istio-podinfo\") pod \"2bb9a4d9-6480-470b-967f-96b99f9ff606\" (UID: \"2bb9a4d9-6480-470b-967f-96b99f9ff606\") " Oct 30 21:50:56.599116 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:56.598997 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bb9a4d9-6480-470b-967f-96b99f9ff606-istio-token\" (UniqueName: \"kubernetes.io/projected/2bb9a4d9-6480-470b-967f-96b99f9ff606-istio-token\") pod \"2bb9a4d9-6480-470b-967f-96b99f9ff606\" (UID: \"2bb9a4d9-6480-470b-967f-96b99f9ff606\") " Oct 30 21:50:56.599116 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:56.599050 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb9a4d9-6480-470b-967f-96b99f9ff606-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb9a4d9-6480-470b-967f-96b99f9ff606-istio-envoy\") pod \"2bb9a4d9-6480-470b-967f-96b99f9ff606\" (UID: \"2bb9a4d9-6480-470b-967f-96b99f9ff606\") " Oct 30 21:50:56.599220 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:56.599120 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb9a4d9-6480-470b-967f-96b99f9ff606-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb9a4d9-6480-470b-967f-96b99f9ff606-workload-socket\") pod \"2bb9a4d9-6480-470b-967f-96b99f9ff606\" (UID: \"2bb9a4d9-6480-470b-967f-96b99f9ff606\") " Oct 30 21:50:56.599220 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:56.599112 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb9a4d9-6480-470b-967f-96b99f9ff606-credential-socket" pod "2bb9a4d9-6480-470b-967f-96b99f9ff606" (UID: "2bb9a4d9-6480-470b-967f-96b99f9ff606"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:56.599220 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:56.599162 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb9a4d9-6480-470b-967f-96b99f9ff606-workload-certs" pod "2bb9a4d9-6480-470b-967f-96b99f9ff606" (UID: "2bb9a4d9-6480-470b-967f-96b99f9ff606"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:56.599220 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:56.599181 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb9a4d9-6480-470b-967f-96b99f9ff606-istio-data" pod "2bb9a4d9-6480-470b-967f-96b99f9ff606" (UID: "2bb9a4d9-6480-470b-967f-96b99f9ff606"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:56.599494 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:56.599432 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb9a4d9-6480-470b-967f-96b99f9ff606-workload-socket" pod "2bb9a4d9-6480-470b-967f-96b99f9ff606" (UID: "2bb9a4d9-6480-470b-967f-96b99f9ff606"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:56.599494 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:56.599473 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bb9a4d9-6480-470b-967f-96b99f9ff606-istiod-ca-cert" pod "2bb9a4d9-6480-470b-967f-96b99f9ff606" (UID: "2bb9a4d9-6480-470b-967f-96b99f9ff606"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:56.599811 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:56.599791 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb9a4d9-6480-470b-967f-96b99f9ff606-credential-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:56.599846 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:56.599819 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb9a4d9-6480-470b-967f-96b99f9ff606-istio-data\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:56.599846 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:56.599835 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb9a4d9-6480-470b-967f-96b99f9ff606-workload-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:56.599918 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:56.599850 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb9a4d9-6480-470b-967f-96b99f9ff606-istiod-ca-cert\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:56.599918 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:56.599866 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb9a4d9-6480-470b-967f-96b99f9ff606-workload-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:56.601176 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:56.601151 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2bb9a4d9-6480-470b-967f-96b99f9ff606-istio-podinfo" pod "2bb9a4d9-6480-470b-967f-96b99f9ff606" (UID: "2bb9a4d9-6480-470b-967f-96b99f9ff606"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:50:56.601301 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:56.601222 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bb9a4d9-6480-470b-967f-96b99f9ff606-istio-token" pod "2bb9a4d9-6480-470b-967f-96b99f9ff606" (UID: "2bb9a4d9-6480-470b-967f-96b99f9ff606"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:56.601301 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:56.601287 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bb9a4d9-6480-470b-967f-96b99f9ff606-kube-api-access-s8fdk" pod "2bb9a4d9-6480-470b-967f-96b99f9ff606" (UID: "2bb9a4d9-6480-470b-967f-96b99f9ff606"). InnerVolumeSpecName "kube-api-access-s8fdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:56.601474 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:56.601460 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb9a4d9-6480-470b-967f-96b99f9ff606-istio-envoy" pod "2bb9a4d9-6480-470b-967f-96b99f9ff606" (UID: "2bb9a4d9-6480-470b-967f-96b99f9ff606"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:56.701283 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:56.701220 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bb9a4d9-6480-470b-967f-96b99f9ff606-istio-token\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:56.701283 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:56.701245 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb9a4d9-6480-470b-967f-96b99f9ff606-istio-envoy\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:56.701283 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:56.701255 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8fdk\" (UniqueName: \"kubernetes.io/projected/2bb9a4d9-6480-470b-967f-96b99f9ff606-kube-api-access-s8fdk\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:56.701283 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:56.701264 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb9a4d9-6480-470b-967f-96b99f9ff606-istio-podinfo\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:57.122796 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:57.122762 4560 generic.go:356] "Generic (PLEG): container finished" podID="2bb9a4d9-6480-470b-967f-96b99f9ff606" containerID="95ec0a84a1a56bce468788fb97dad5fe4d70f7ff4d155f9f804e617f90b63930" exitCode=0 Oct 30 21:50:57.122991 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:57.122821 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" Oct 30 21:50:57.122991 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:57.122842 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" event={"ID":"2bb9a4d9-6480-470b-967f-96b99f9ff606","Type":"ContainerDied","Data":"95ec0a84a1a56bce468788fb97dad5fe4d70f7ff4d155f9f804e617f90b63930"} Oct 30 21:50:57.122991 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:57.122899 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx" event={"ID":"2bb9a4d9-6480-470b-967f-96b99f9ff606","Type":"ContainerDied","Data":"d07b5227190a3aacfa6e5da86317a330cc2c6e79f005fe1e39e01da91bced79b"} Oct 30 21:50:57.122991 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:57.122921 4560 scope.go:122] "RemoveContainer" containerID="95ec0a84a1a56bce468788fb97dad5fe4d70f7ff4d155f9f804e617f90b63930" Oct 30 21:50:57.135033 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:57.135014 4560 scope.go:122] "RemoveContainer" containerID="95ec0a84a1a56bce468788fb97dad5fe4d70f7ff4d155f9f804e617f90b63930" Oct 30 21:50:57.135286 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:50:57.135264 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95ec0a84a1a56bce468788fb97dad5fe4d70f7ff4d155f9f804e617f90b63930\": container with ID starting with 95ec0a84a1a56bce468788fb97dad5fe4d70f7ff4d155f9f804e617f90b63930 not found: ID does not exist" containerID="95ec0a84a1a56bce468788fb97dad5fe4d70f7ff4d155f9f804e617f90b63930" Oct 30 21:50:57.135379 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:57.135293 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95ec0a84a1a56bce468788fb97dad5fe4d70f7ff4d155f9f804e617f90b63930"} err="failed to get container status \"95ec0a84a1a56bce468788fb97dad5fe4d70f7ff4d155f9f804e617f90b63930\": rpc error: code = NotFound desc = could not find container \"95ec0a84a1a56bce468788fb97dad5fe4d70f7ff4d155f9f804e617f90b63930\": container with ID starting with 95ec0a84a1a56bce468788fb97dad5fe4d70f7ff4d155f9f804e617f90b63930 not found: ID does not exist" Oct 30 21:50:57.327646 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:57.327615 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx"] Oct 30 21:50:57.330677 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:57.330656 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-66gjx"] Oct 30 21:50:58.432720 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:58.432689 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-364e1518-655d99d7d-5fmrs"] Oct 30 21:50:58.548932 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:58.548890 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-364e1518-8c6bd84bc-ffns9"] Oct 30 21:50:58.576184 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:50:58.576152 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bb9a4d9-6480-470b-967f-96b99f9ff606" path="/var/lib/kubelet/pods/2bb9a4d9-6480-470b-967f-96b99f9ff606/volumes" Oct 30 21:51:03.433463 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:03.433416 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-364e1518-655d99d7d-5fmrs" podUID="dc1d2d01-9d75-4e02-820d-19d1cbaa18c9" containerName="payload-processing" containerID="cri-o://e2691544660f173bea2a847c2846f4eedced4bc846a13e0248f085e0396e192e" gracePeriod=25 Oct 30 21:51:03.549936 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:03.549895 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-364e1518-8c6bd84bc-ffns9" podUID="3fdf9c46-b0e6-409e-b102-81800c96e24c" containerName="payload-pre-processing" containerID="cri-o://ea127caeac8bed97eaf236c3b367699c02fe50e539a6f805c7b8df2648d3cf4f" gracePeriod=25 Oct 30 21:51:03.724978 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:03.724950 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-364e1518-655d99d7d-5fmrs" Oct 30 21:51:03.825731 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:03.825230 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc1d2d01-9d75-4e02-820d-19d1cbaa18c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc1d2d01-9d75-4e02-820d-19d1cbaa18c9-plugins-config-volume\") pod \"dc1d2d01-9d75-4e02-820d-19d1cbaa18c9\" (UID: \"dc1d2d01-9d75-4e02-820d-19d1cbaa18c9\") " Oct 30 21:51:03.825731 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:03.825292 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc1d2d01-9d75-4e02-820d-19d1cbaa18c9-kube-api-access-k4cfz\" (UniqueName: \"kubernetes.io/projected/dc1d2d01-9d75-4e02-820d-19d1cbaa18c9-kube-api-access-k4cfz\") pod \"dc1d2d01-9d75-4e02-820d-19d1cbaa18c9\" (UID: \"dc1d2d01-9d75-4e02-820d-19d1cbaa18c9\") " Oct 30 21:51:03.827144 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:03.827102 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc1d2d01-9d75-4e02-820d-19d1cbaa18c9-plugins-config-volume" pod "dc1d2d01-9d75-4e02-820d-19d1cbaa18c9" (UID: "dc1d2d01-9d75-4e02-820d-19d1cbaa18c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:03.925657 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:03.925619 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc1d2d01-9d75-4e02-820d-19d1cbaa18c9-kube-api-access-k4cfz" pod "dc1d2d01-9d75-4e02-820d-19d1cbaa18c9" (UID: "dc1d2d01-9d75-4e02-820d-19d1cbaa18c9"). InnerVolumeSpecName "kube-api-access-k4cfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:03.928053 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:03.927979 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc1d2d01-9d75-4e02-820d-19d1cbaa18c9-plugins-config-volume\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:03.928053 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:03.928013 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4cfz\" (UniqueName: \"kubernetes.io/projected/dc1d2d01-9d75-4e02-820d-19d1cbaa18c9-kube-api-access-k4cfz\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:03.938151 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:03.938121 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-364e1518-8c6bd84bc-ffns9" Oct 30 21:51:04.045027 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:04.044991 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fdf9c46-b0e6-409e-b102-81800c96e24c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fdf9c46-b0e6-409e-b102-81800c96e24c-plugins-config-volume\") pod \"3fdf9c46-b0e6-409e-b102-81800c96e24c\" (UID: \"3fdf9c46-b0e6-409e-b102-81800c96e24c\") " Oct 30 21:51:04.052518 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:04.046213 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fdf9c46-b0e6-409e-b102-81800c96e24c-plugins-config-volume" pod "3fdf9c46-b0e6-409e-b102-81800c96e24c" (UID: "3fdf9c46-b0e6-409e-b102-81800c96e24c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:04.052770 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:04.052750 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fdf9c46-b0e6-409e-b102-81800c96e24c-kube-api-access-tsx6v\" (UniqueName: \"kubernetes.io/projected/3fdf9c46-b0e6-409e-b102-81800c96e24c-kube-api-access-tsx6v\") pod \"3fdf9c46-b0e6-409e-b102-81800c96e24c\" (UID: \"3fdf9c46-b0e6-409e-b102-81800c96e24c\") " Oct 30 21:51:04.070039 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:04.059303 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fdf9c46-b0e6-409e-b102-81800c96e24c-plugins-config-volume\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:04.072782 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:04.072735 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fdf9c46-b0e6-409e-b102-81800c96e24c-kube-api-access-tsx6v" pod "3fdf9c46-b0e6-409e-b102-81800c96e24c" (UID: "3fdf9c46-b0e6-409e-b102-81800c96e24c"). InnerVolumeSpecName "kube-api-access-tsx6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:04.171463 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:04.171434 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsx6v\" (UniqueName: \"kubernetes.io/projected/3fdf9c46-b0e6-409e-b102-81800c96e24c-kube-api-access-tsx6v\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:04.228220 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:04.228184 4560 generic.go:356] "Generic (PLEG): container finished" podID="dc1d2d01-9d75-4e02-820d-19d1cbaa18c9" containerID="e2691544660f173bea2a847c2846f4eedced4bc846a13e0248f085e0396e192e" exitCode=0 Oct 30 21:51:04.228428 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:04.228229 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-364e1518-655d99d7d-5fmrs" Oct 30 21:51:04.228428 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:04.228267 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-364e1518-655d99d7d-5fmrs" event={"ID":"dc1d2d01-9d75-4e02-820d-19d1cbaa18c9","Type":"ContainerDied","Data":"e2691544660f173bea2a847c2846f4eedced4bc846a13e0248f085e0396e192e"} Oct 30 21:51:04.228428 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:04.228322 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-364e1518-655d99d7d-5fmrs" event={"ID":"dc1d2d01-9d75-4e02-820d-19d1cbaa18c9","Type":"ContainerDied","Data":"b4425e8467cb83f41ae3bfb28511ac462c5e919cc5f7fe5f2c952a55fe6d9da3"} Oct 30 21:51:04.228428 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:04.228385 4560 scope.go:122] "RemoveContainer" containerID="e2691544660f173bea2a847c2846f4eedced4bc846a13e0248f085e0396e192e" Oct 30 21:51:04.229986 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:04.229961 4560 generic.go:356] "Generic (PLEG): container finished" podID="3fdf9c46-b0e6-409e-b102-81800c96e24c" containerID="ea127caeac8bed97eaf236c3b367699c02fe50e539a6f805c7b8df2648d3cf4f" exitCode=0 Oct 30 21:51:04.230114 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:04.229997 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-364e1518-8c6bd84bc-ffns9" event={"ID":"3fdf9c46-b0e6-409e-b102-81800c96e24c","Type":"ContainerDied","Data":"ea127caeac8bed97eaf236c3b367699c02fe50e539a6f805c7b8df2648d3cf4f"} Oct 30 21:51:04.230114 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:04.230020 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-364e1518-8c6bd84bc-ffns9" Oct 30 21:51:04.230114 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:04.230054 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-364e1518-8c6bd84bc-ffns9" event={"ID":"3fdf9c46-b0e6-409e-b102-81800c96e24c","Type":"ContainerDied","Data":"88ecafeda0c58ed3af5204f66d9556004d397db22cd20e56838e0682af444e9b"} Oct 30 21:51:04.239574 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:04.239492 4560 scope.go:122] "RemoveContainer" containerID="e2691544660f173bea2a847c2846f4eedced4bc846a13e0248f085e0396e192e" Oct 30 21:51:04.239837 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:51:04.239813 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2691544660f173bea2a847c2846f4eedced4bc846a13e0248f085e0396e192e\": container with ID starting with e2691544660f173bea2a847c2846f4eedced4bc846a13e0248f085e0396e192e not found: ID does not exist" containerID="e2691544660f173bea2a847c2846f4eedced4bc846a13e0248f085e0396e192e" Oct 30 21:51:04.239902 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:04.239850 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2691544660f173bea2a847c2846f4eedced4bc846a13e0248f085e0396e192e"} err="failed to get container status \"e2691544660f173bea2a847c2846f4eedced4bc846a13e0248f085e0396e192e\": rpc error: code = NotFound desc = could not find container \"e2691544660f173bea2a847c2846f4eedced4bc846a13e0248f085e0396e192e\": container with ID starting with e2691544660f173bea2a847c2846f4eedced4bc846a13e0248f085e0396e192e not found: ID does not exist" Oct 30 21:51:04.239902 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:04.239873 4560 scope.go:122] "RemoveContainer" containerID="ea127caeac8bed97eaf236c3b367699c02fe50e539a6f805c7b8df2648d3cf4f" Oct 30 21:51:04.248698 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:04.248677 4560 scope.go:122] "RemoveContainer" containerID="ea127caeac8bed97eaf236c3b367699c02fe50e539a6f805c7b8df2648d3cf4f" Oct 30 21:51:04.248949 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:51:04.248931 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea127caeac8bed97eaf236c3b367699c02fe50e539a6f805c7b8df2648d3cf4f\": container with ID starting with ea127caeac8bed97eaf236c3b367699c02fe50e539a6f805c7b8df2648d3cf4f not found: ID does not exist" containerID="ea127caeac8bed97eaf236c3b367699c02fe50e539a6f805c7b8df2648d3cf4f" Oct 30 21:51:04.249023 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:04.248956 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea127caeac8bed97eaf236c3b367699c02fe50e539a6f805c7b8df2648d3cf4f"} err="failed to get container status \"ea127caeac8bed97eaf236c3b367699c02fe50e539a6f805c7b8df2648d3cf4f\": rpc error: code = NotFound desc = could not find container \"ea127caeac8bed97eaf236c3b367699c02fe50e539a6f805c7b8df2648d3cf4f\": container with ID starting with ea127caeac8bed97eaf236c3b367699c02fe50e539a6f805c7b8df2648d3cf4f not found: ID does not exist" Oct 30 21:51:04.388026 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:04.387933 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-364e1518-8c6bd84bc-ffns9"] Oct 30 21:51:04.391050 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:04.391026 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-364e1518-8c6bd84bc-ffns9"] Oct 30 21:51:04.494082 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:04.494046 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-364e1518-655d99d7d-5fmrs"] Oct 30 21:51:04.500686 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:04.500662 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-364e1518-655d99d7d-5fmrs"] Oct 30 21:51:04.588820 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:04.588782 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fdf9c46-b0e6-409e-b102-81800c96e24c" path="/var/lib/kubelet/pods/3fdf9c46-b0e6-409e-b102-81800c96e24c/volumes" Oct 30 21:51:04.589198 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:04.589181 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc1d2d01-9d75-4e02-820d-19d1cbaa18c9" path="/var/lib/kubelet/pods/dc1d2d01-9d75-4e02-820d-19d1cbaa18c9/volumes" Oct 30 21:51:11.048434 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:11.048401 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-157f1e1f-2nx6l"] Oct 30 21:51:11.095309 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:11.095281 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-157f1e1f-2nx6l"] Oct 30 21:51:12.576835 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:12.576799 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dbb0932-e12b-4768-861b-4832d25dd952" path="/var/lib/kubelet/pods/0dbb0932-e12b-4768-861b-4832d25dd952/volumes" Oct 30 21:51:15.059022 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:15.058990 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f22ef26-vww99"] Oct 30 21:51:15.106560 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:15.106525 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f22ef26-vww99"] Oct 30 21:51:16.598176 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:16.598138 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2711f3d0-ec20-4184-bc59-0d3a36c6abf3" path="/var/lib/kubelet/pods/2711f3d0-ec20-4184-bc59-0d3a36c6abf3/volumes" Oct 30 21:51:19.550838 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:19.550802 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:51:19.684830 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:51:19.684790 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:51:23.012121 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:23.012076 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-655c1d85-gm7jv"] Oct 30 21:51:23.012766 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:23.012596 4560 cpu_manager.go:414] "containerMap: removing container" podUID="3fdf9c46-b0e6-409e-b102-81800c96e24c" containerName="payload-pre-processing" Oct 30 21:51:23.012766 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:23.012609 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fdf9c46-b0e6-409e-b102-81800c96e24c" containerName="payload-pre-processing" Oct 30 21:51:23.012766 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:23.012645 4560 cpu_manager.go:414] "containerMap: removing container" podUID="dc1d2d01-9d75-4e02-820d-19d1cbaa18c9" containerName="payload-processing" Oct 30 21:51:23.012766 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:23.012651 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc1d2d01-9d75-4e02-820d-19d1cbaa18c9" containerName="payload-processing" Oct 30 21:51:23.012766 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:23.012663 4560 cpu_manager.go:414] "containerMap: removing container" podUID="60e92ada-e6f3-4b6a-b82f-728481f30aff" containerName="revoke-keys" Oct 30 21:51:23.012766 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:23.012668 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60e92ada-e6f3-4b6a-b82f-728481f30aff" containerName="revoke-keys" Oct 30 21:51:23.012766 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:23.012676 4560 cpu_manager.go:414] "containerMap: removing container" podUID="2bb9a4d9-6480-470b-967f-96b99f9ff606" containerName="istio-proxy" Oct 30 21:51:23.012766 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:23.012681 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bb9a4d9-6480-470b-967f-96b99f9ff606" containerName="istio-proxy" Oct 30 21:51:23.012766 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:23.012769 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="60e92ada-e6f3-4b6a-b82f-728481f30aff" containerName="revoke-keys" Oct 30 21:51:23.012766 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:23.012776 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bb9a4d9-6480-470b-967f-96b99f9ff606" containerName="istio-proxy" Oct 30 21:51:23.013271 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:23.012784 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fdf9c46-b0e6-409e-b102-81800c96e24c" containerName="payload-pre-processing" Oct 30 21:51:23.013271 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:23.012796 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc1d2d01-9d75-4e02-820d-19d1cbaa18c9" containerName="payload-processing" Oct 30 21:51:23.017483 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:23.017461 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655c1d85-gm7jv" Oct 30 21:51:23.117757 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:23.117732 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:51:23.117757 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:23.117730 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:51:23.118130 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:23.118109 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:51:23.118202 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:23.118150 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:51:23.121378 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:23.121361 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-655c1d85-gm7jv"] Oct 30 21:51:23.174678 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:23.174643 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca7f9e19-a746-42c9-89fd-3361c8684774-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-655c1d85-gm7jv\" (UID: \"ca7f9e19-a746-42c9-89fd-3361c8684774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655c1d85-gm7jv" Oct 30 21:51:23.174854 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:23.174824 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca7f9e19-a746-42c9-89fd-3361c8684774-maas-api-xks-ca\") pod \"maas-api-revoke-sub-655c1d85-gm7jv\" (UID: \"ca7f9e19-a746-42c9-89fd-3361c8684774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655c1d85-gm7jv" Oct 30 21:51:23.174968 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:23.174944 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcxsw\" (UniqueName: \"kubernetes.io/projected/ca7f9e19-a746-42c9-89fd-3361c8684774-kube-api-access-dcxsw\") pod \"maas-api-revoke-sub-655c1d85-gm7jv\" (UID: \"ca7f9e19-a746-42c9-89fd-3361c8684774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655c1d85-gm7jv" Oct 30 21:51:23.276950 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:23.276879 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcxsw\" (UniqueName: \"kubernetes.io/projected/ca7f9e19-a746-42c9-89fd-3361c8684774-kube-api-access-dcxsw\") pod \"maas-api-revoke-sub-655c1d85-gm7jv\" (UID: \"ca7f9e19-a746-42c9-89fd-3361c8684774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655c1d85-gm7jv" Oct 30 21:51:23.277106 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:23.276980 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca7f9e19-a746-42c9-89fd-3361c8684774-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-655c1d85-gm7jv\" (UID: \"ca7f9e19-a746-42c9-89fd-3361c8684774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655c1d85-gm7jv" Oct 30 21:51:23.277171 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:23.277159 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca7f9e19-a746-42c9-89fd-3361c8684774-maas-api-xks-ca\") pod \"maas-api-revoke-sub-655c1d85-gm7jv\" (UID: \"ca7f9e19-a746-42c9-89fd-3361c8684774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655c1d85-gm7jv" Oct 30 21:51:23.277692 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:23.277663 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca7f9e19-a746-42c9-89fd-3361c8684774-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-655c1d85-gm7jv\" (UID: \"ca7f9e19-a746-42c9-89fd-3361c8684774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655c1d85-gm7jv" Oct 30 21:51:23.279679 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:23.279657 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca7f9e19-a746-42c9-89fd-3361c8684774-maas-api-xks-ca\") pod \"maas-api-revoke-sub-655c1d85-gm7jv\" (UID: \"ca7f9e19-a746-42c9-89fd-3361c8684774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655c1d85-gm7jv" Oct 30 21:51:23.340628 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:23.340594 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcxsw\" (UniqueName: \"kubernetes.io/projected/ca7f9e19-a746-42c9-89fd-3361c8684774-kube-api-access-dcxsw\") pod \"maas-api-revoke-sub-655c1d85-gm7jv\" (UID: \"ca7f9e19-a746-42c9-89fd-3361c8684774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655c1d85-gm7jv" Oct 30 21:51:23.641060 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:23.640975 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655c1d85-gm7jv" Oct 30 21:51:23.913472 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:23.913382 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-40bb78-6d7d4d7644-jwwbv"] Oct 30 21:51:23.920510 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:23.920484 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-40bb78-6d7d4d7644-jwwbv" Oct 30 21:51:23.971048 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:23.971015 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-40bb78-6d7d4d7644-jwwbv"] Oct 30 21:51:23.972370 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:23.972140 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-40bb78\"" Oct 30 21:51:23.972553 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:23.972257 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-40bb78-dockercfg-497fz\"" Oct 30 21:51:24.077928 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:24.071290 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/345a9ec6-062e-4402-83fc-15adac6800bd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-40bb78-6d7d4d7644-jwwbv\" (UID: \"345a9ec6-062e-4402-83fc-15adac6800bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-40bb78-6d7d4d7644-jwwbv" Oct 30 21:51:24.077928 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:24.071425 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8bzp\" (UniqueName: \"kubernetes.io/projected/345a9ec6-062e-4402-83fc-15adac6800bd-kube-api-access-p8bzp\") pod \"payload-processing-e2e-shared-a-w5-40bb78-6d7d4d7644-jwwbv\" (UID: \"345a9ec6-062e-4402-83fc-15adac6800bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-40bb78-6d7d4d7644-jwwbv" Oct 30 21:51:24.212022 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:24.211937 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/345a9ec6-062e-4402-83fc-15adac6800bd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-40bb78-6d7d4d7644-jwwbv\" (UID: \"345a9ec6-062e-4402-83fc-15adac6800bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-40bb78-6d7d4d7644-jwwbv" Oct 30 21:51:24.212530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:24.212501 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8bzp\" (UniqueName: \"kubernetes.io/projected/345a9ec6-062e-4402-83fc-15adac6800bd-kube-api-access-p8bzp\") pod \"payload-processing-e2e-shared-a-w5-40bb78-6d7d4d7644-jwwbv\" (UID: \"345a9ec6-062e-4402-83fc-15adac6800bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-40bb78-6d7d4d7644-jwwbv" Oct 30 21:51:24.213217 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:24.213187 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/345a9ec6-062e-4402-83fc-15adac6800bd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-40bb78-6d7d4d7644-jwwbv\" (UID: \"345a9ec6-062e-4402-83fc-15adac6800bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-40bb78-6d7d4d7644-jwwbv" Oct 30 21:51:24.270960 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:24.270929 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8bzp\" (UniqueName: \"kubernetes.io/projected/345a9ec6-062e-4402-83fc-15adac6800bd-kube-api-access-p8bzp\") pod \"payload-processing-e2e-shared-a-w5-40bb78-6d7d4d7644-jwwbv\" (UID: \"345a9ec6-062e-4402-83fc-15adac6800bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-40bb78-6d7d4d7644-jwwbv" Oct 30 21:51:24.381079 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:24.381045 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-655c1d85-gm7jv"] Oct 30 21:51:24.386092 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:51:24.386065 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca7f9e19_a746_42c9_89fd_3361c8684774.slice/crio-ae516ccd76008351968fecbdbded9cf794b32766fd05df55065288babba0412d WatchSource:0}: Error finding container ae516ccd76008351968fecbdbded9cf794b32766fd05df55065288babba0412d: Status 404 returned error can't find the container with id ae516ccd76008351968fecbdbded9cf794b32766fd05df55065288babba0412d Oct 30 21:51:24.526921 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:24.526888 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655c1d85-gm7jv" event={"ID":"ca7f9e19-a746-42c9-89fd-3361c8684774","Type":"ContainerStarted","Data":"ae516ccd76008351968fecbdbded9cf794b32766fd05df55065288babba0412d"} Oct 30 21:51:24.546149 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:24.546129 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-40bb78-6d7d4d7644-jwwbv" Oct 30 21:51:24.927719 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:24.927633 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-40bb78-6d7d4d7644-jwwbv"] Oct 30 21:51:25.536036 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:25.535999 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-40bb78-6d7d4d7644-jwwbv" event={"ID":"345a9ec6-062e-4402-83fc-15adac6800bd","Type":"ContainerStarted","Data":"9e97a8c9bbf328276d7147f509d20b591fa5833094c6d49562c80de24c9ea5db"} Oct 30 21:51:25.536519 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:25.536092 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-40bb78-6d7d4d7644-jwwbv" Oct 30 21:51:25.536519 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:25.536118 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-40bb78-6d7d4d7644-jwwbv" event={"ID":"345a9ec6-062e-4402-83fc-15adac6800bd","Type":"ContainerStarted","Data":"50bab75277461bba0ebcf20d56fd438b8f32de22f5028f3a4e6db005afe25845"} Oct 30 21:51:25.537643 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:25.537618 4560 generic.go:356] "Generic (PLEG): container finished" podID="ca7f9e19-a746-42c9-89fd-3361c8684774" containerID="dcd7e38bb903266af81cb601f4ebe51a2134ebe599279be937de978977efd42c" exitCode=0 Oct 30 21:51:25.537761 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:25.537683 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655c1d85-gm7jv" event={"ID":"ca7f9e19-a746-42c9-89fd-3361c8684774","Type":"ContainerDied","Data":"dcd7e38bb903266af81cb601f4ebe51a2134ebe599279be937de978977efd42c"} Oct 30 21:51:25.649026 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:25.648928 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-40bb78-6d7d4d7644-jwwbv" podStartSLOduration=2.648915569 podStartE2EDuration="2.648915569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:25.646998408 +0000 UTC m=+2261.201698185" watchObservedRunningTime="2026-10-30 21:51:25.648915569 +0000 UTC m=+2261.203615359" Oct 30 21:51:26.053545 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:26.053482 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e24f13ff-z9rch"] Oct 30 21:51:26.057768 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:26.057742 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e24f13ff-z9rch"] Oct 30 21:51:26.577953 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:26.577923 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e22781aa-a00f-4608-956a-a5c81d03641b" path="/var/lib/kubelet/pods/e22781aa-a00f-4608-956a-a5c81d03641b/volumes" Oct 30 21:51:26.690732 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:26.690708 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655c1d85-gm7jv" Oct 30 21:51:26.748296 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:26.748267 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca7f9e19-a746-42c9-89fd-3361c8684774-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca7f9e19-a746-42c9-89fd-3361c8684774-maas-api-xks-ca\") pod \"ca7f9e19-a746-42c9-89fd-3361c8684774\" (UID: \"ca7f9e19-a746-42c9-89fd-3361c8684774\") " Oct 30 21:51:26.748500 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:26.748419 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca7f9e19-a746-42c9-89fd-3361c8684774-kube-api-access-dcxsw\" (UniqueName: \"kubernetes.io/projected/ca7f9e19-a746-42c9-89fd-3361c8684774-kube-api-access-dcxsw\") pod \"ca7f9e19-a746-42c9-89fd-3361c8684774\" (UID: \"ca7f9e19-a746-42c9-89fd-3361c8684774\") " Oct 30 21:51:26.748500 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:26.748471 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca7f9e19-a746-42c9-89fd-3361c8684774-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca7f9e19-a746-42c9-89fd-3361c8684774-maas-api-openshift-ca\") pod \"ca7f9e19-a746-42c9-89fd-3361c8684774\" (UID: \"ca7f9e19-a746-42c9-89fd-3361c8684774\") " Oct 30 21:51:26.748882 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:26.748853 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca7f9e19-a746-42c9-89fd-3361c8684774-maas-api-openshift-ca" pod "ca7f9e19-a746-42c9-89fd-3361c8684774" (UID: "ca7f9e19-a746-42c9-89fd-3361c8684774"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:26.749144 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:26.749127 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca7f9e19-a746-42c9-89fd-3361c8684774-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:26.750812 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:26.750785 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca7f9e19-a746-42c9-89fd-3361c8684774-maas-api-xks-ca" pod "ca7f9e19-a746-42c9-89fd-3361c8684774" (UID: "ca7f9e19-a746-42c9-89fd-3361c8684774"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:26.750908 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:26.750814 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca7f9e19-a746-42c9-89fd-3361c8684774-kube-api-access-dcxsw" pod "ca7f9e19-a746-42c9-89fd-3361c8684774" (UID: "ca7f9e19-a746-42c9-89fd-3361c8684774"). InnerVolumeSpecName "kube-api-access-dcxsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:26.850856 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:26.850769 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca7f9e19-a746-42c9-89fd-3361c8684774-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:26.850856 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:26.850802 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcxsw\" (UniqueName: \"kubernetes.io/projected/ca7f9e19-a746-42c9-89fd-3361c8684774-kube-api-access-dcxsw\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:27.552707 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:27.552677 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655c1d85-gm7jv" Oct 30 21:51:27.552883 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:27.552721 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655c1d85-gm7jv" event={"ID":"ca7f9e19-a746-42c9-89fd-3361c8684774","Type":"ContainerDied","Data":"ae516ccd76008351968fecbdbded9cf794b32766fd05df55065288babba0412d"} Oct 30 21:51:27.552883 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:27.552769 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae516ccd76008351968fecbdbded9cf794b32766fd05df55065288babba0412d" Oct 30 21:51:35.723369 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:35.717273 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fbdbd87a-mmptb"] Oct 30 21:51:35.779177 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:35.779137 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fbdbd87a-mmptb"] Oct 30 21:51:36.547634 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:36.547606 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-40bb78-6d7d4d7644-jwwbv" Oct 30 21:51:36.577036 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:36.576997 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60e92ada-e6f3-4b6a-b82f-728481f30aff" path="/var/lib/kubelet/pods/60e92ada-e6f3-4b6a-b82f-728481f30aff/volumes" Oct 30 21:51:36.849045 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:36.848949 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-364e1518-nc9sd"] Oct 30 21:51:36.854388 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:36.854356 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-364e1518-nc9sd"] Oct 30 21:51:38.579628 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:38.579590 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f9a82e0-3d49-40ef-a1fc-405f179cbf77" path="/var/lib/kubelet/pods/2f9a82e0-3d49-40ef-a1fc-405f179cbf77/volumes" Oct 30 21:51:41.909170 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:41.909138 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-bfdf30b9/e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc"] Oct 30 21:51:41.909657 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:41.909643 4560 cpu_manager.go:414] "containerMap: removing container" podUID="ca7f9e19-a746-42c9-89fd-3361c8684774" containerName="revoke-keys" Oct 30 21:51:41.909700 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:41.909658 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca7f9e19-a746-42c9-89fd-3361c8684774" containerName="revoke-keys" Oct 30 21:51:41.909776 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:41.909767 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca7f9e19-a746-42c9-89fd-3361c8684774" containerName="revoke-keys" Oct 30 21:51:41.913060 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:41.913044 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bfdf30b9/e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc" Oct 30 21:51:41.964302 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:41.964275 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-bfdf30b9\"/\"openshift-service-ca.crt\"" Oct 30 21:51:41.965507 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:41.964538 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-bfdf30b9\"/\"kube-root-ca.crt\"" Oct 30 21:51:41.983567 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:41.983541 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90966441-80bf-4bc5-8e68-4dea00d9f779-tmp-dir\") pod \"e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc\" (UID: \"90966441-80bf-4bc5-8e68-4dea00d9f779\") " pod="ai-tenant-e2e-mt-bfdf30b9/e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc" Oct 30 21:51:41.983724 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:41.983708 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90966441-80bf-4bc5-8e68-4dea00d9f779-model-cache\") pod \"e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc\" (UID: \"90966441-80bf-4bc5-8e68-4dea00d9f779\") " pod="ai-tenant-e2e-mt-bfdf30b9/e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc" Oct 30 21:51:41.983841 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:41.983823 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90966441-80bf-4bc5-8e68-4dea00d9f779-dshm\") pod \"e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc\" (UID: \"90966441-80bf-4bc5-8e68-4dea00d9f779\") " pod="ai-tenant-e2e-mt-bfdf30b9/e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc" Oct 30 21:51:41.983883 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:41.983868 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90966441-80bf-4bc5-8e68-4dea00d9f779-tls-certs\") pod \"e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc\" (UID: \"90966441-80bf-4bc5-8e68-4dea00d9f779\") " pod="ai-tenant-e2e-mt-bfdf30b9/e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc" Oct 30 21:51:41.983918 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:41.983900 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90966441-80bf-4bc5-8e68-4dea00d9f779-home\") pod \"e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc\" (UID: \"90966441-80bf-4bc5-8e68-4dea00d9f779\") " pod="ai-tenant-e2e-mt-bfdf30b9/e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc" Oct 30 21:51:41.984063 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:41.984046 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fdx6\" (UniqueName: \"kubernetes.io/projected/90966441-80bf-4bc5-8e68-4dea00d9f779-kube-api-access-4fdx6\") pod \"e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc\" (UID: \"90966441-80bf-4bc5-8e68-4dea00d9f779\") " pod="ai-tenant-e2e-mt-bfdf30b9/e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc" Oct 30 21:51:42.014366 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:42.013652 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-bfdf30b9\"/\"default-dockercfg-nnkdc\"" Oct 30 21:51:42.014366 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:42.013777 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-bfdf30b9\"/\"e2e-lifecycle-model-bfdf30b9-kserve-self-signed-certs\"" Oct 30 21:51:42.061026 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:42.060922 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-bfdf30b9/e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc"] Oct 30 21:51:42.152855 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:42.152800 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90966441-80bf-4bc5-8e68-4dea00d9f779-home\") pod \"e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc\" (UID: \"90966441-80bf-4bc5-8e68-4dea00d9f779\") " pod="ai-tenant-e2e-mt-bfdf30b9/e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc" Oct 30 21:51:42.172465 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:42.153515 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90966441-80bf-4bc5-8e68-4dea00d9f779-home\") pod \"e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc\" (UID: \"90966441-80bf-4bc5-8e68-4dea00d9f779\") " pod="ai-tenant-e2e-mt-bfdf30b9/e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc" Oct 30 21:51:42.172863 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:42.172830 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fdx6\" (UniqueName: \"kubernetes.io/projected/90966441-80bf-4bc5-8e68-4dea00d9f779-kube-api-access-4fdx6\") pod \"e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc\" (UID: \"90966441-80bf-4bc5-8e68-4dea00d9f779\") " pod="ai-tenant-e2e-mt-bfdf30b9/e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc" Oct 30 21:51:42.173460 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:42.173437 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90966441-80bf-4bc5-8e68-4dea00d9f779-tmp-dir\") pod \"e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc\" (UID: \"90966441-80bf-4bc5-8e68-4dea00d9f779\") " pod="ai-tenant-e2e-mt-bfdf30b9/e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc" Oct 30 21:51:42.174132 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:42.174102 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90966441-80bf-4bc5-8e68-4dea00d9f779-model-cache\") pod \"e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc\" (UID: \"90966441-80bf-4bc5-8e68-4dea00d9f779\") " pod="ai-tenant-e2e-mt-bfdf30b9/e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc" Oct 30 21:51:42.174836 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:42.174810 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90966441-80bf-4bc5-8e68-4dea00d9f779-tmp-dir\") pod \"e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc\" (UID: \"90966441-80bf-4bc5-8e68-4dea00d9f779\") " pod="ai-tenant-e2e-mt-bfdf30b9/e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc" Oct 30 21:51:42.175102 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:42.174694 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90966441-80bf-4bc5-8e68-4dea00d9f779-model-cache\") pod \"e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc\" (UID: \"90966441-80bf-4bc5-8e68-4dea00d9f779\") " pod="ai-tenant-e2e-mt-bfdf30b9/e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc" Oct 30 21:51:42.175912 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:42.175675 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90966441-80bf-4bc5-8e68-4dea00d9f779-dshm\") pod \"e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc\" (UID: \"90966441-80bf-4bc5-8e68-4dea00d9f779\") " pod="ai-tenant-e2e-mt-bfdf30b9/e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc" Oct 30 21:51:42.176086 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:42.176074 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90966441-80bf-4bc5-8e68-4dea00d9f779-tls-certs\") pod \"e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc\" (UID: \"90966441-80bf-4bc5-8e68-4dea00d9f779\") " pod="ai-tenant-e2e-mt-bfdf30b9/e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc" Oct 30 21:51:42.180068 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:42.179378 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90966441-80bf-4bc5-8e68-4dea00d9f779-dshm\") pod \"e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc\" (UID: \"90966441-80bf-4bc5-8e68-4dea00d9f779\") " pod="ai-tenant-e2e-mt-bfdf30b9/e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc" Oct 30 21:51:42.185304 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:42.185282 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90966441-80bf-4bc5-8e68-4dea00d9f779-tls-certs\") pod \"e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc\" (UID: \"90966441-80bf-4bc5-8e68-4dea00d9f779\") " pod="ai-tenant-e2e-mt-bfdf30b9/e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc" Oct 30 21:51:42.234022 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:42.233984 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fdx6\" (UniqueName: \"kubernetes.io/projected/90966441-80bf-4bc5-8e68-4dea00d9f779-kube-api-access-4fdx6\") pod \"e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc\" (UID: \"90966441-80bf-4bc5-8e68-4dea00d9f779\") " pod="ai-tenant-e2e-mt-bfdf30b9/e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc" Oct 30 21:51:42.525363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:42.525304 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bfdf30b9/e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc" Oct 30 21:51:42.936308 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:42.936228 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-bfdf30b9/e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc"] Oct 30 21:51:42.940791 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:51:42.940759 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90966441_80bf_4bc5_8e68_4dea00d9f779.slice/crio-6f742e70a474155f8f9cbdd2a4c8d9393527ea10d4c6de117201bf84882d9b04 WatchSource:0}: Error finding container 6f742e70a474155f8f9cbdd2a4c8d9393527ea10d4c6de117201bf84882d9b04: Status 404 returned error can't find the container with id 6f742e70a474155f8f9cbdd2a4c8d9393527ea10d4c6de117201bf84882d9b04 Oct 30 21:51:43.772686 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:43.772650 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bfdf30b9/e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc" event={"ID":"90966441-80bf-4bc5-8e68-4dea00d9f779","Type":"ContainerStarted","Data":"b1c4594f3a30dcbfb66af51cfbfbc5300f208b15e8d641ca2999e5a644223b81"} Oct 30 21:51:43.772686 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:43.772693 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bfdf30b9/e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc" event={"ID":"90966441-80bf-4bc5-8e68-4dea00d9f779","Type":"ContainerStarted","Data":"6f742e70a474155f8f9cbdd2a4c8d9393527ea10d4c6de117201bf84882d9b04"} Oct 30 21:51:43.885997 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:43.885941 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-bfdf30b9/e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc" podStartSLOduration=2.885927396 podStartE2EDuration="2.885927396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:43.883431909 +0000 UTC m=+2279.438131689" watchObservedRunningTime="2026-10-30 21:51:43.885927396 +0000 UTC m=+2279.440627172" Oct 30 21:51:44.049094 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:44.049016 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b78ca65b-hbdf2"] Oct 30 21:51:44.053729 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:44.053705 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b78ca65b-hbdf2"] Oct 30 21:51:44.525857 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:44.525827 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-bfdf30b9/e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc" Oct 30 21:51:44.610620 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:44.610568 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd250b4b-880a-47b1-bd3c-7843bc9d0f28" path="/var/lib/kubelet/pods/fd250b4b-880a-47b1-bd3c-7843bc9d0f28/volumes" Oct 30 21:51:44.838804 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:44.838729 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq"] Oct 30 21:51:44.838956 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:44.838913 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" podUID="5c0221dd-a19a-464f-93af-996a4a53a62a" containerName="istio-proxy" containerID="cri-o://c864772955893be48a5dee27e687d4a2e92e16f3295b72a1874d0a116483e381" gracePeriod=30 Oct 30 21:51:46.050285 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:46.050253 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd1f2444-7ss99"] Oct 30 21:51:46.096486 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:46.096455 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd1f2444-7ss99"] Oct 30 21:51:46.317932 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:46.317862 4560 scope.go:122] "RemoveContainer" containerID="7bb4dc5ee6d7c667bb8b520957a0c46fb0be949333a9ad9ce2db6fee1b83872a" Oct 30 21:51:46.327723 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:46.327701 4560 scope.go:122] "RemoveContainer" containerID="eec5a9005f5115f1d5e8226609ddda9c5fa6955aad6bd5a5d9e8fababb1fd0cd" Oct 30 21:51:46.339092 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:46.339066 4560 scope.go:122] "RemoveContainer" containerID="8183db8753050c39913c87ff7768213c7edd7587a91695e0928fc601351ea1be" Oct 30 21:51:46.347546 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:46.347530 4560 scope.go:122] "RemoveContainer" containerID="32bf939d57ac2e126f84b717ea48117229fdf9498e669754183332fda92c3588" Oct 30 21:51:46.355324 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:46.355308 4560 scope.go:122] "RemoveContainer" containerID="be46a34984458e2c8b231b9c9b3012acfc8c6e220f3a2815e42d0b235e3a513f" Oct 30 21:51:46.363142 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:46.363114 4560 scope.go:122] "RemoveContainer" containerID="24ff079eaa0b550cbeec030a978b17af6ee1c29db652091cbc0c9c6c6741722a" Oct 30 21:51:46.371242 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:46.371228 4560 scope.go:122] "RemoveContainer" containerID="32a9401f562311c7b2adff2eed2206f78d3f869632910a1a1af42aa0f1da9ce5" Oct 30 21:51:46.378810 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:46.378793 4560 scope.go:122] "RemoveContainer" containerID="964755de842d9cc9e9e74e310e3ea127ac02f60b0127cfe298d72f1b31061497" Oct 30 21:51:46.386276 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:46.386261 4560 scope.go:122] "RemoveContainer" containerID="0e4ad9b07bec93e19a8b3c854212292836d678d0b6494538611bf1aae2b46ff6" Oct 30 21:51:46.393803 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:46.393788 4560 scope.go:122] "RemoveContainer" containerID="e5d238fc7af74f2cd0e036b58b77302d9ae8726d90df3f92815826c01b58938d" Oct 30 21:51:46.401471 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:46.401453 4560 scope.go:122] "RemoveContainer" containerID="a8c0e748b87d96333b296539d4b5af68acdaf320812f30cc3ee8ec5aeb74b9dc" Oct 30 21:51:46.409115 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:46.409098 4560 scope.go:122] "RemoveContainer" containerID="acf7e506cdb282b58ea74ec14dfd77caeca0d7d289dfbb91d98bc22aaa1bb88a" Oct 30 21:51:46.562026 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:46.562001 4560 patch_prober.go:29] interesting pod/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:51:46.562157 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:46.562060 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" podUID="5c0221dd-a19a-464f-93af-996a4a53a62a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:51:46.577174 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:46.577122 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806" path="/var/lib/kubelet/pods/a9dc400f-0aa4-4ccf-a93d-81f3c8bf4806/volumes" Oct 30 21:51:49.684668 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:51:49.684630 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:51:49.980846 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:49.980817 4560 generic.go:356] "Generic (PLEG): container finished" podID="5c0221dd-a19a-464f-93af-996a4a53a62a" containerID="c864772955893be48a5dee27e687d4a2e92e16f3295b72a1874d0a116483e381" exitCode=0 Oct 30 21:51:49.981005 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:49.980889 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" event={"ID":"5c0221dd-a19a-464f-93af-996a4a53a62a","Type":"ContainerDied","Data":"c864772955893be48a5dee27e687d4a2e92e16f3295b72a1874d0a116483e381"} Oct 30 21:51:49.981005 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:49.980962 4560 scope.go:122] "RemoveContainer" containerID="3b3013fa31a125cb8facbd94d31e33324f30129daa8160f1fa385b10c36fbc6e" Oct 30 21:51:50.049486 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:50.048836 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62bafda8-wtgtq"] Oct 30 21:51:50.095371 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:50.095314 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62bafda8-wtgtq"] Oct 30 21:51:50.119834 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:50.119812 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" Oct 30 21:51:50.318412 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:50.299465 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c0221dd-a19a-464f-93af-996a4a53a62a-istio-token\" (UniqueName: \"kubernetes.io/projected/5c0221dd-a19a-464f-93af-996a4a53a62a-istio-token\") pod \"5c0221dd-a19a-464f-93af-996a4a53a62a\" (UID: \"5c0221dd-a19a-464f-93af-996a4a53a62a\") " Oct 30 21:51:50.318412 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:50.299549 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c0221dd-a19a-464f-93af-996a4a53a62a-kube-api-access-b247p\" (UniqueName: \"kubernetes.io/projected/5c0221dd-a19a-464f-93af-996a4a53a62a-kube-api-access-b247p\") pod \"5c0221dd-a19a-464f-93af-996a4a53a62a\" (UID: \"5c0221dd-a19a-464f-93af-996a4a53a62a\") " Oct 30 21:51:50.318412 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:50.299605 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c0221dd-a19a-464f-93af-996a4a53a62a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0221dd-a19a-464f-93af-996a4a53a62a-credential-socket\") pod \"5c0221dd-a19a-464f-93af-996a4a53a62a\" (UID: \"5c0221dd-a19a-464f-93af-996a4a53a62a\") " Oct 30 21:51:50.318412 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:50.299698 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c0221dd-a19a-464f-93af-996a4a53a62a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c0221dd-a19a-464f-93af-996a4a53a62a-istio-data\") pod \"5c0221dd-a19a-464f-93af-996a4a53a62a\" (UID: \"5c0221dd-a19a-464f-93af-996a4a53a62a\") " Oct 30 21:51:50.318412 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:50.299758 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c0221dd-a19a-464f-93af-996a4a53a62a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c0221dd-a19a-464f-93af-996a4a53a62a-istio-podinfo\") pod \"5c0221dd-a19a-464f-93af-996a4a53a62a\" (UID: \"5c0221dd-a19a-464f-93af-996a4a53a62a\") " Oct 30 21:51:50.318412 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:50.299795 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c0221dd-a19a-464f-93af-996a4a53a62a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c0221dd-a19a-464f-93af-996a4a53a62a-istiod-ca-cert\") pod \"5c0221dd-a19a-464f-93af-996a4a53a62a\" (UID: \"5c0221dd-a19a-464f-93af-996a4a53a62a\") " Oct 30 21:51:50.318412 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:50.299876 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c0221dd-a19a-464f-93af-996a4a53a62a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0221dd-a19a-464f-93af-996a4a53a62a-workload-socket\") pod \"5c0221dd-a19a-464f-93af-996a4a53a62a\" (UID: \"5c0221dd-a19a-464f-93af-996a4a53a62a\") " Oct 30 21:51:50.318412 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:50.299965 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c0221dd-a19a-464f-93af-996a4a53a62a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c0221dd-a19a-464f-93af-996a4a53a62a-workload-certs\") pod \"5c0221dd-a19a-464f-93af-996a4a53a62a\" (UID: \"5c0221dd-a19a-464f-93af-996a4a53a62a\") " Oct 30 21:51:50.318412 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:50.300019 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c0221dd-a19a-464f-93af-996a4a53a62a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c0221dd-a19a-464f-93af-996a4a53a62a-istio-envoy\") pod \"5c0221dd-a19a-464f-93af-996a4a53a62a\" (UID: \"5c0221dd-a19a-464f-93af-996a4a53a62a\") " Oct 30 21:51:50.318412 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:50.303792 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c0221dd-a19a-464f-93af-996a4a53a62a-istio-envoy" pod "5c0221dd-a19a-464f-93af-996a4a53a62a" (UID: "5c0221dd-a19a-464f-93af-996a4a53a62a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:50.341683 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:50.340998 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c0221dd-a19a-464f-93af-996a4a53a62a-istio-data" pod "5c0221dd-a19a-464f-93af-996a4a53a62a" (UID: "5c0221dd-a19a-464f-93af-996a4a53a62a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:50.341683 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:50.341161 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c0221dd-a19a-464f-93af-996a4a53a62a-credential-socket" pod "5c0221dd-a19a-464f-93af-996a4a53a62a" (UID: "5c0221dd-a19a-464f-93af-996a4a53a62a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:50.341683 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:50.341307 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c0221dd-a19a-464f-93af-996a4a53a62a-workload-socket" pod "5c0221dd-a19a-464f-93af-996a4a53a62a" (UID: "5c0221dd-a19a-464f-93af-996a4a53a62a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:50.341683 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:50.341487 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c0221dd-a19a-464f-93af-996a4a53a62a-workload-certs" pod "5c0221dd-a19a-464f-93af-996a4a53a62a" (UID: "5c0221dd-a19a-464f-93af-996a4a53a62a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:50.341683 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:50.341652 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c0221dd-a19a-464f-93af-996a4a53a62a-istio-token" pod "5c0221dd-a19a-464f-93af-996a4a53a62a" (UID: "5c0221dd-a19a-464f-93af-996a4a53a62a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:50.341683 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:50.341652 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c0221dd-a19a-464f-93af-996a4a53a62a-istiod-ca-cert" pod "5c0221dd-a19a-464f-93af-996a4a53a62a" (UID: "5c0221dd-a19a-464f-93af-996a4a53a62a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:50.398422 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:50.380466 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c0221dd-a19a-464f-93af-996a4a53a62a-istio-podinfo" pod "5c0221dd-a19a-464f-93af-996a4a53a62a" (UID: "5c0221dd-a19a-464f-93af-996a4a53a62a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:50.415448 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:50.400703 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c0221dd-a19a-464f-93af-996a4a53a62a-kube-api-access-b247p" pod "5c0221dd-a19a-464f-93af-996a4a53a62a" (UID: "5c0221dd-a19a-464f-93af-996a4a53a62a"). InnerVolumeSpecName "kube-api-access-b247p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:50.415448 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:50.405420 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c0221dd-a19a-464f-93af-996a4a53a62a-istio-podinfo\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:50.415448 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:50.405453 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c0221dd-a19a-464f-93af-996a4a53a62a-istiod-ca-cert\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:50.415448 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:50.405467 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0221dd-a19a-464f-93af-996a4a53a62a-workload-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:50.415448 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:50.405479 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c0221dd-a19a-464f-93af-996a4a53a62a-workload-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:50.415448 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:50.405492 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c0221dd-a19a-464f-93af-996a4a53a62a-istio-envoy\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:50.415448 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:50.405504 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c0221dd-a19a-464f-93af-996a4a53a62a-istio-token\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:50.415448 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:50.405518 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b247p\" (UniqueName: \"kubernetes.io/projected/5c0221dd-a19a-464f-93af-996a4a53a62a-kube-api-access-b247p\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:50.415448 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:50.405532 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0221dd-a19a-464f-93af-996a4a53a62a-credential-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:50.415448 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:50.405545 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c0221dd-a19a-464f-93af-996a4a53a62a-istio-data\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:50.576976 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:50.576904 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39" path="/var/lib/kubelet/pods/5a6f8fc4-1854-4ef1-aa67-0cb4c1627e39/volumes" Oct 30 21:51:50.988847 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:50.988819 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" event={"ID":"5c0221dd-a19a-464f-93af-996a4a53a62a","Type":"ContainerDied","Data":"173b822715915167cb38f5898a576822d50a8432396ff4e7178e73d705541818"} Oct 30 21:51:50.988847 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:50.988833 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq" Oct 30 21:51:50.989315 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:50.988872 4560 scope.go:122] "RemoveContainer" containerID="c864772955893be48a5dee27e687d4a2e92e16f3295b72a1874d0a116483e381" Oct 30 21:51:51.197947 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:51.197871 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq"] Oct 30 21:51:51.242104 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:51.242074 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-364e1518-openshift-default-5ddfff4c97-t72jq"] Oct 30 21:51:52.051761 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:52.051719 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9217b2b5-hxxgv"] Oct 30 21:51:52.054421 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:52.054387 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9217b2b5-hxxgv"] Oct 30 21:51:52.526063 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:52.526035 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-bfdf30b9/e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc" Oct 30 21:51:52.538657 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:52.538634 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-bfdf30b9/e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc" Oct 30 21:51:52.577455 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:52.577426 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c0221dd-a19a-464f-93af-996a4a53a62a" path="/var/lib/kubelet/pods/5c0221dd-a19a-464f-93af-996a4a53a62a/volumes" Oct 30 21:51:52.577996 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:52.577981 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d25781de-5282-4af7-8199-a1fa695f19ba" path="/var/lib/kubelet/pods/d25781de-5282-4af7-8199-a1fa695f19ba/volumes" Oct 30 21:51:53.220361 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:53.218566 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-bfdf30b9/e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc" Oct 30 21:51:58.598041 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:58.597954 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h"] Oct 30 21:51:58.598779 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:58.598757 4560 cpu_manager.go:414] "containerMap: removing container" podUID="5c0221dd-a19a-464f-93af-996a4a53a62a" containerName="istio-proxy" Oct 30 21:51:58.598859 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:58.598780 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c0221dd-a19a-464f-93af-996a4a53a62a" containerName="istio-proxy" Oct 30 21:51:58.598859 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:58.598813 4560 cpu_manager.go:414] "containerMap: removing container" podUID="5c0221dd-a19a-464f-93af-996a4a53a62a" containerName="istio-proxy" Oct 30 21:51:58.598859 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:58.598823 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c0221dd-a19a-464f-93af-996a4a53a62a" containerName="istio-proxy" Oct 30 21:51:58.599012 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:58.598956 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c0221dd-a19a-464f-93af-996a4a53a62a" containerName="istio-proxy" Oct 30 21:51:58.599012 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:58.598975 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c0221dd-a19a-464f-93af-996a4a53a62a" containerName="istio-proxy" Oct 30 21:51:58.634422 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:58.633778 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" Oct 30 21:51:58.688216 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:58.686812 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-2aa268-openshift-default-dockercfg-d7z9x\"" Oct 30 21:51:58.732364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:58.727894 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a8c5ad5-8b50-4795-9596-d951de399015-istio-envoy\") pod \"e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h\" (UID: \"7a8c5ad5-8b50-4795-9596-d951de399015\") " pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" Oct 30 21:51:58.748542 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:58.748497 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a8c5ad5-8b50-4795-9596-d951de399015-istiod-ca-cert\") pod \"e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h\" (UID: \"7a8c5ad5-8b50-4795-9596-d951de399015\") " pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" Oct 30 21:51:58.748805 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:58.748785 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a8c5ad5-8b50-4795-9596-d951de399015-istio-token\") pod \"e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h\" (UID: \"7a8c5ad5-8b50-4795-9596-d951de399015\") " pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" Oct 30 21:51:58.748935 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:58.748921 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a8c5ad5-8b50-4795-9596-d951de399015-istio-podinfo\") pod \"e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h\" (UID: \"7a8c5ad5-8b50-4795-9596-d951de399015\") " pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" Oct 30 21:51:58.749033 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:58.749021 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a8c5ad5-8b50-4795-9596-d951de399015-workload-certs\") pod \"e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h\" (UID: \"7a8c5ad5-8b50-4795-9596-d951de399015\") " pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" Oct 30 21:51:58.749150 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:58.749137 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8c5ad5-8b50-4795-9596-d951de399015-credential-socket\") pod \"e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h\" (UID: \"7a8c5ad5-8b50-4795-9596-d951de399015\") " pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" Oct 30 21:51:58.749296 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:58.749281 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8c5ad5-8b50-4795-9596-d951de399015-workload-socket\") pod \"e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h\" (UID: \"7a8c5ad5-8b50-4795-9596-d951de399015\") " pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" Oct 30 21:51:58.749416 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:58.749401 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a8c5ad5-8b50-4795-9596-d951de399015-istio-data\") pod \"e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h\" (UID: \"7a8c5ad5-8b50-4795-9596-d951de399015\") " pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" Oct 30 21:51:58.749848 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:58.749829 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcwd7\" (UniqueName: \"kubernetes.io/projected/7a8c5ad5-8b50-4795-9596-d951de399015-kube-api-access-wcwd7\") pod \"e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h\" (UID: \"7a8c5ad5-8b50-4795-9596-d951de399015\") " pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" Oct 30 21:51:58.775960 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:58.775914 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h"] Oct 30 21:51:58.854491 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:58.854192 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a8c5ad5-8b50-4795-9596-d951de399015-istio-token\") pod \"e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h\" (UID: \"7a8c5ad5-8b50-4795-9596-d951de399015\") " pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" Oct 30 21:51:58.854491 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:58.854254 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a8c5ad5-8b50-4795-9596-d951de399015-istio-podinfo\") pod \"e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h\" (UID: \"7a8c5ad5-8b50-4795-9596-d951de399015\") " pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" Oct 30 21:51:58.854491 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:58.854282 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a8c5ad5-8b50-4795-9596-d951de399015-workload-certs\") pod \"e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h\" (UID: \"7a8c5ad5-8b50-4795-9596-d951de399015\") " pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" Oct 30 21:51:58.854491 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:58.854323 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8c5ad5-8b50-4795-9596-d951de399015-credential-socket\") pod \"e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h\" (UID: \"7a8c5ad5-8b50-4795-9596-d951de399015\") " pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" Oct 30 21:51:58.854491 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:58.854401 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8c5ad5-8b50-4795-9596-d951de399015-workload-socket\") pod \"e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h\" (UID: \"7a8c5ad5-8b50-4795-9596-d951de399015\") " pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" Oct 30 21:51:58.854491 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:58.854424 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a8c5ad5-8b50-4795-9596-d951de399015-istio-data\") pod \"e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h\" (UID: \"7a8c5ad5-8b50-4795-9596-d951de399015\") " pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" Oct 30 21:51:58.854960 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:58.854606 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcwd7\" (UniqueName: \"kubernetes.io/projected/7a8c5ad5-8b50-4795-9596-d951de399015-kube-api-access-wcwd7\") pod \"e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h\" (UID: \"7a8c5ad5-8b50-4795-9596-d951de399015\") " pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" Oct 30 21:51:58.854960 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:58.854748 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a8c5ad5-8b50-4795-9596-d951de399015-istio-envoy\") pod \"e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h\" (UID: \"7a8c5ad5-8b50-4795-9596-d951de399015\") " pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" Oct 30 21:51:58.854960 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:58.854826 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a8c5ad5-8b50-4795-9596-d951de399015-istiod-ca-cert\") pod \"e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h\" (UID: \"7a8c5ad5-8b50-4795-9596-d951de399015\") " pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" Oct 30 21:51:58.925370 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:58.855744 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a8c5ad5-8b50-4795-9596-d951de399015-istiod-ca-cert\") pod \"e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h\" (UID: \"7a8c5ad5-8b50-4795-9596-d951de399015\") " pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" Oct 30 21:51:58.925643 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:58.868763 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8c5ad5-8b50-4795-9596-d951de399015-credential-socket\") pod \"e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h\" (UID: \"7a8c5ad5-8b50-4795-9596-d951de399015\") " pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" Oct 30 21:51:58.935438 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:58.870209 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a8c5ad5-8b50-4795-9596-d951de399015-workload-certs\") pod \"e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h\" (UID: \"7a8c5ad5-8b50-4795-9596-d951de399015\") " pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" Oct 30 21:51:58.935641 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:58.870423 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a8c5ad5-8b50-4795-9596-d951de399015-istio-data\") pod \"e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h\" (UID: \"7a8c5ad5-8b50-4795-9596-d951de399015\") " pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" Oct 30 21:51:58.935736 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:58.870613 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8c5ad5-8b50-4795-9596-d951de399015-workload-socket\") pod \"e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h\" (UID: \"7a8c5ad5-8b50-4795-9596-d951de399015\") " pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" Oct 30 21:51:58.943448 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:58.880381 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a8c5ad5-8b50-4795-9596-d951de399015-istio-envoy\") pod \"e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h\" (UID: \"7a8c5ad5-8b50-4795-9596-d951de399015\") " pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" Oct 30 21:51:58.949453 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:58.931717 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a8c5ad5-8b50-4795-9596-d951de399015-istio-podinfo\") pod \"e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h\" (UID: \"7a8c5ad5-8b50-4795-9596-d951de399015\") " pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" Oct 30 21:51:58.976369 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:58.961609 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a8c5ad5-8b50-4795-9596-d951de399015-istio-token\") pod \"e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h\" (UID: \"7a8c5ad5-8b50-4795-9596-d951de399015\") " pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" Oct 30 21:51:58.976369 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:58.962185 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcwd7\" (UniqueName: \"kubernetes.io/projected/7a8c5ad5-8b50-4795-9596-d951de399015-kube-api-access-wcwd7\") pod \"e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h\" (UID: \"7a8c5ad5-8b50-4795-9596-d951de399015\") " pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" Oct 30 21:51:59.080435 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:59.039186 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" Oct 30 21:51:59.549215 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:59.549184 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h"] Oct 30 21:51:59.553524 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:51:59.553489 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a8c5ad5_8b50_4795_9596_d951de399015.slice/crio-00a95dbdf1102f62e85004d83980c174685bdcd82f64abee736e575656d7304b WatchSource:0}: Error finding container 00a95dbdf1102f62e85004d83980c174685bdcd82f64abee736e575656d7304b: Status 404 returned error can't find the container with id 00a95dbdf1102f62e85004d83980c174685bdcd82f64abee736e575656d7304b Oct 30 21:51:59.556518 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:59.556472 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:51:59.556615 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:59.556567 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:51:59.556615 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:51:59.556609 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:52:00.231757 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:00.231721 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" event={"ID":"7a8c5ad5-8b50-4795-9596-d951de399015","Type":"ContainerStarted","Data":"ddf18f19c908cd1807a1db3b7a9a0ff05d35e44448bf0556ee0ea9d41d3c2276"} Oct 30 21:52:00.232146 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:00.231773 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" event={"ID":"7a8c5ad5-8b50-4795-9596-d951de399015","Type":"ContainerStarted","Data":"00a95dbdf1102f62e85004d83980c174685bdcd82f64abee736e575656d7304b"} Oct 30 21:52:00.395612 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:00.395555 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" podStartSLOduration=2.395536857 podStartE2EDuration="2.395536857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:00.393998574 +0000 UTC m=+2295.948698372" watchObservedRunningTime="2026-10-30 21:52:00.395536857 +0000 UTC m=+2295.950236636" Oct 30 21:52:01.064366 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:01.053673 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" Oct 30 21:52:01.146078 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:01.146047 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" Oct 30 21:52:01.396059 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:01.395920 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" Oct 30 21:52:01.404441 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:01.404326 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" Oct 30 21:52:10.834781 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:10.834746 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29f04460-2xvqd"] Oct 30 21:52:10.838630 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:10.838610 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f04460-2xvqd" Oct 30 21:52:10.890373 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:10.890317 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:52:10.903375 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:10.901267 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be68ac41-bb28-46e6-acf3-b880a29513d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29f04460-2xvqd\" (UID: \"be68ac41-bb28-46e6-acf3-b880a29513d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f04460-2xvqd" Oct 30 21:52:10.903375 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:10.901478 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqx52\" (UniqueName: \"kubernetes.io/projected/be68ac41-bb28-46e6-acf3-b880a29513d3-kube-api-access-mqx52\") pod \"maas-api-revoke-sub-29f04460-2xvqd\" (UID: \"be68ac41-bb28-46e6-acf3-b880a29513d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f04460-2xvqd" Oct 30 21:52:10.903375 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:10.901518 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be68ac41-bb28-46e6-acf3-b880a29513d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29f04460-2xvqd\" (UID: \"be68ac41-bb28-46e6-acf3-b880a29513d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f04460-2xvqd" Oct 30 21:52:10.937482 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:10.937458 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:52:10.937482 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:10.937454 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:52:10.937660 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:10.937534 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:52:10.941827 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:10.941803 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29f04460-2xvqd"] Oct 30 21:52:11.003402 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:11.003374 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqx52\" (UniqueName: \"kubernetes.io/projected/be68ac41-bb28-46e6-acf3-b880a29513d3-kube-api-access-mqx52\") pod \"maas-api-revoke-sub-29f04460-2xvqd\" (UID: \"be68ac41-bb28-46e6-acf3-b880a29513d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f04460-2xvqd" Oct 30 21:52:11.003566 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:11.003416 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be68ac41-bb28-46e6-acf3-b880a29513d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29f04460-2xvqd\" (UID: \"be68ac41-bb28-46e6-acf3-b880a29513d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f04460-2xvqd" Oct 30 21:52:11.003566 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:11.003532 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be68ac41-bb28-46e6-acf3-b880a29513d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29f04460-2xvqd\" (UID: \"be68ac41-bb28-46e6-acf3-b880a29513d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f04460-2xvqd" Oct 30 21:52:11.004241 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:11.004215 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be68ac41-bb28-46e6-acf3-b880a29513d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29f04460-2xvqd\" (UID: \"be68ac41-bb28-46e6-acf3-b880a29513d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f04460-2xvqd" Oct 30 21:52:11.005933 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:11.005908 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be68ac41-bb28-46e6-acf3-b880a29513d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29f04460-2xvqd\" (UID: \"be68ac41-bb28-46e6-acf3-b880a29513d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f04460-2xvqd" Oct 30 21:52:11.061971 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:11.061940 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqx52\" (UniqueName: \"kubernetes.io/projected/be68ac41-bb28-46e6-acf3-b880a29513d3-kube-api-access-mqx52\") pod \"maas-api-revoke-sub-29f04460-2xvqd\" (UID: \"be68ac41-bb28-46e6-acf3-b880a29513d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f04460-2xvqd" Oct 30 21:52:11.152926 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:11.152851 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f04460-2xvqd" Oct 30 21:52:11.533684 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:11.533648 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29f04460-2xvqd"] Oct 30 21:52:11.540283 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:52:11.540256 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe68ac41_bb28_46e6_acf3_b880a29513d3.slice/crio-96bf2f1720bbff1af7884fd4d1bda5501ed3b3fba51738ff3efb8a80121dc28a WatchSource:0}: Error finding container 96bf2f1720bbff1af7884fd4d1bda5501ed3b3fba51738ff3efb8a80121dc28a: Status 404 returned error can't find the container with id 96bf2f1720bbff1af7884fd4d1bda5501ed3b3fba51738ff3efb8a80121dc28a Oct 30 21:52:11.709922 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:11.709886 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f04460-2xvqd" event={"ID":"be68ac41-bb28-46e6-acf3-b880a29513d3","Type":"ContainerStarted","Data":"96bf2f1720bbff1af7884fd4d1bda5501ed3b3fba51738ff3efb8a80121dc28a"} Oct 30 21:52:12.773586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:12.773069 4560 generic.go:356] "Generic (PLEG): container finished" podID="be68ac41-bb28-46e6-acf3-b880a29513d3" containerID="46bca71a1bfc89a84503754a38f887d8963047d714ff552a2c5047b2e8908305" exitCode=0 Oct 30 21:52:12.773586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:12.773148 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f04460-2xvqd" event={"ID":"be68ac41-bb28-46e6-acf3-b880a29513d3","Type":"ContainerDied","Data":"46bca71a1bfc89a84503754a38f887d8963047d714ff552a2c5047b2e8908305"} Oct 30 21:52:13.929566 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:13.929543 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f04460-2xvqd" Oct 30 21:52:14.083937 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:14.083856 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be68ac41-bb28-46e6-acf3-b880a29513d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be68ac41-bb28-46e6-acf3-b880a29513d3-maas-api-openshift-ca\") pod \"be68ac41-bb28-46e6-acf3-b880a29513d3\" (UID: \"be68ac41-bb28-46e6-acf3-b880a29513d3\") " Oct 30 21:52:14.083937 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:14.083898 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be68ac41-bb28-46e6-acf3-b880a29513d3-kube-api-access-mqx52\" (UniqueName: \"kubernetes.io/projected/be68ac41-bb28-46e6-acf3-b880a29513d3-kube-api-access-mqx52\") pod \"be68ac41-bb28-46e6-acf3-b880a29513d3\" (UID: \"be68ac41-bb28-46e6-acf3-b880a29513d3\") " Oct 30 21:52:14.084169 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:14.084062 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be68ac41-bb28-46e6-acf3-b880a29513d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be68ac41-bb28-46e6-acf3-b880a29513d3-maas-api-xks-ca\") pod \"be68ac41-bb28-46e6-acf3-b880a29513d3\" (UID: \"be68ac41-bb28-46e6-acf3-b880a29513d3\") " Oct 30 21:52:14.084756 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:14.084599 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be68ac41-bb28-46e6-acf3-b880a29513d3-maas-api-openshift-ca" pod "be68ac41-bb28-46e6-acf3-b880a29513d3" (UID: "be68ac41-bb28-46e6-acf3-b880a29513d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:14.085042 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:14.084883 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be68ac41-bb28-46e6-acf3-b880a29513d3-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:14.086777 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:14.086753 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be68ac41-bb28-46e6-acf3-b880a29513d3-kube-api-access-mqx52" pod "be68ac41-bb28-46e6-acf3-b880a29513d3" (UID: "be68ac41-bb28-46e6-acf3-b880a29513d3"). InnerVolumeSpecName "kube-api-access-mqx52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:14.086969 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:14.086951 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be68ac41-bb28-46e6-acf3-b880a29513d3-maas-api-xks-ca" pod "be68ac41-bb28-46e6-acf3-b880a29513d3" (UID: "be68ac41-bb28-46e6-acf3-b880a29513d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:52:14.185957 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:14.185928 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqx52\" (UniqueName: \"kubernetes.io/projected/be68ac41-bb28-46e6-acf3-b880a29513d3-kube-api-access-mqx52\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:14.185957 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:14.185953 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be68ac41-bb28-46e6-acf3-b880a29513d3-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:14.787024 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:14.786993 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f04460-2xvqd" Oct 30 21:52:14.787169 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:14.787018 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f04460-2xvqd" event={"ID":"be68ac41-bb28-46e6-acf3-b880a29513d3","Type":"ContainerDied","Data":"96bf2f1720bbff1af7884fd4d1bda5501ed3b3fba51738ff3efb8a80121dc28a"} Oct 30 21:52:14.787169 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:14.787065 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96bf2f1720bbff1af7884fd4d1bda5501ed3b3fba51738ff3efb8a80121dc28a" Oct 30 21:52:19.694233 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:52:19.694193 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:52:35.545499 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:35.545468 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq"] Oct 30 21:52:35.546239 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:35.546213 4560 cpu_manager.go:414] "containerMap: removing container" podUID="be68ac41-bb28-46e6-acf3-b880a29513d3" containerName="revoke-keys" Oct 30 21:52:35.546239 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:35.546236 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be68ac41-bb28-46e6-acf3-b880a29513d3" containerName="revoke-keys" Oct 30 21:52:35.546469 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:35.546390 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="be68ac41-bb28-46e6-acf3-b880a29513d3" containerName="revoke-keys" Oct 30 21:52:35.550034 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:35.550014 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" Oct 30 21:52:35.713909 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:35.713872 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c73ef128-5710-4654-a155-4b9083ab04bb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-m2sxq\" (UID: \"c73ef128-5710-4654-a155-4b9083ab04bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" Oct 30 21:52:35.714076 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:35.713950 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73ef128-5710-4654-a155-4b9083ab04bb-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-m2sxq\" (UID: \"c73ef128-5710-4654-a155-4b9083ab04bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" Oct 30 21:52:35.714166 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:35.714140 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7p29\" (UniqueName: \"kubernetes.io/projected/c73ef128-5710-4654-a155-4b9083ab04bb-kube-api-access-h7p29\") pod \"maas-default-gateway-openshift-default-67bf986f99-m2sxq\" (UID: \"c73ef128-5710-4654-a155-4b9083ab04bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" Oct 30 21:52:35.714208 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:35.714181 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c73ef128-5710-4654-a155-4b9083ab04bb-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-m2sxq\" (UID: \"c73ef128-5710-4654-a155-4b9083ab04bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" Oct 30 21:52:35.714332 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:35.714311 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c73ef128-5710-4654-a155-4b9083ab04bb-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-m2sxq\" (UID: \"c73ef128-5710-4654-a155-4b9083ab04bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" Oct 30 21:52:35.714467 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:35.714394 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c73ef128-5710-4654-a155-4b9083ab04bb-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-m2sxq\" (UID: \"c73ef128-5710-4654-a155-4b9083ab04bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" Oct 30 21:52:35.714528 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:35.714504 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c73ef128-5710-4654-a155-4b9083ab04bb-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-m2sxq\" (UID: \"c73ef128-5710-4654-a155-4b9083ab04bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" Oct 30 21:52:35.714606 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:35.714588 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73ef128-5710-4654-a155-4b9083ab04bb-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-m2sxq\" (UID: \"c73ef128-5710-4654-a155-4b9083ab04bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" Oct 30 21:52:35.714665 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:35.714619 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c73ef128-5710-4654-a155-4b9083ab04bb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-m2sxq\" (UID: \"c73ef128-5710-4654-a155-4b9083ab04bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" Oct 30 21:52:35.755287 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:35.755258 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq"] Oct 30 21:52:35.816273 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:35.816201 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7p29\" (UniqueName: \"kubernetes.io/projected/c73ef128-5710-4654-a155-4b9083ab04bb-kube-api-access-h7p29\") pod \"maas-default-gateway-openshift-default-67bf986f99-m2sxq\" (UID: \"c73ef128-5710-4654-a155-4b9083ab04bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" Oct 30 21:52:35.816273 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:35.816241 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c73ef128-5710-4654-a155-4b9083ab04bb-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-m2sxq\" (UID: \"c73ef128-5710-4654-a155-4b9083ab04bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" Oct 30 21:52:35.816521 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:35.816294 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c73ef128-5710-4654-a155-4b9083ab04bb-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-m2sxq\" (UID: \"c73ef128-5710-4654-a155-4b9083ab04bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" Oct 30 21:52:35.816521 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:35.816315 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c73ef128-5710-4654-a155-4b9083ab04bb-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-m2sxq\" (UID: \"c73ef128-5710-4654-a155-4b9083ab04bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" Oct 30 21:52:35.816521 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:35.816367 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c73ef128-5710-4654-a155-4b9083ab04bb-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-m2sxq\" (UID: \"c73ef128-5710-4654-a155-4b9083ab04bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" Oct 30 21:52:35.816521 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:35.816401 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73ef128-5710-4654-a155-4b9083ab04bb-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-m2sxq\" (UID: \"c73ef128-5710-4654-a155-4b9083ab04bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" Oct 30 21:52:35.816521 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:35.816427 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c73ef128-5710-4654-a155-4b9083ab04bb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-m2sxq\" (UID: \"c73ef128-5710-4654-a155-4b9083ab04bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" Oct 30 21:52:35.816762 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:35.816548 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c73ef128-5710-4654-a155-4b9083ab04bb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-m2sxq\" (UID: \"c73ef128-5710-4654-a155-4b9083ab04bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" Oct 30 21:52:35.816762 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:35.816582 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73ef128-5710-4654-a155-4b9083ab04bb-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-m2sxq\" (UID: \"c73ef128-5710-4654-a155-4b9083ab04bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" Oct 30 21:52:35.816762 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:35.816737 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c73ef128-5710-4654-a155-4b9083ab04bb-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-m2sxq\" (UID: \"c73ef128-5710-4654-a155-4b9083ab04bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" Oct 30 21:52:35.816923 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:35.816838 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73ef128-5710-4654-a155-4b9083ab04bb-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-m2sxq\" (UID: \"c73ef128-5710-4654-a155-4b9083ab04bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" Oct 30 21:52:35.816923 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:35.816886 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c73ef128-5710-4654-a155-4b9083ab04bb-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-m2sxq\" (UID: \"c73ef128-5710-4654-a155-4b9083ab04bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" Oct 30 21:52:35.817004 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:35.816925 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73ef128-5710-4654-a155-4b9083ab04bb-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-m2sxq\" (UID: \"c73ef128-5710-4654-a155-4b9083ab04bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" Oct 30 21:52:35.817215 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:35.817193 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c73ef128-5710-4654-a155-4b9083ab04bb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-m2sxq\" (UID: \"c73ef128-5710-4654-a155-4b9083ab04bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" Oct 30 21:52:35.818804 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:35.818781 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c73ef128-5710-4654-a155-4b9083ab04bb-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-m2sxq\" (UID: \"c73ef128-5710-4654-a155-4b9083ab04bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" Oct 30 21:52:35.818934 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:35.818916 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c73ef128-5710-4654-a155-4b9083ab04bb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-m2sxq\" (UID: \"c73ef128-5710-4654-a155-4b9083ab04bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" Oct 30 21:52:35.874991 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:35.874957 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7p29\" (UniqueName: \"kubernetes.io/projected/c73ef128-5710-4654-a155-4b9083ab04bb-kube-api-access-h7p29\") pod \"maas-default-gateway-openshift-default-67bf986f99-m2sxq\" (UID: \"c73ef128-5710-4654-a155-4b9083ab04bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" Oct 30 21:52:35.875305 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:35.875282 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c73ef128-5710-4654-a155-4b9083ab04bb-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-m2sxq\" (UID: \"c73ef128-5710-4654-a155-4b9083ab04bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" Oct 30 21:52:36.163882 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:36.163800 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" Oct 30 21:52:36.615498 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:36.615461 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq"] Oct 30 21:52:36.620574 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:52:36.620547 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc73ef128_5710_4654_a155_4b9083ab04bb.slice/crio-107833a71101f91913d933a984a3d905061c5cbb615e5b7b5ff439f16701acf4 WatchSource:0}: Error finding container 107833a71101f91913d933a984a3d905061c5cbb615e5b7b5ff439f16701acf4: Status 404 returned error can't find the container with id 107833a71101f91913d933a984a3d905061c5cbb615e5b7b5ff439f16701acf4 Oct 30 21:52:36.623404 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:36.623363 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:52:36.623508 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:36.623446 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:52:36.623508 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:36.623487 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:52:37.010997 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:37.010967 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" event={"ID":"c73ef128-5710-4654-a155-4b9083ab04bb","Type":"ContainerStarted","Data":"09bf98af6bf3e8052476c01a62b0f2cd36249d56a8667505fe0078c67e8afaa9"} Oct 30 21:52:37.011200 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:37.011019 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" event={"ID":"c73ef128-5710-4654-a155-4b9083ab04bb","Type":"ContainerStarted","Data":"107833a71101f91913d933a984a3d905061c5cbb615e5b7b5ff439f16701acf4"} Oct 30 21:52:37.164614 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:37.164582 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" Oct 30 21:52:37.165844 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:37.165820 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-m2sxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 21:52:37.165970 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:37.165869 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" podUID="c73ef128-5710-4654-a155-4b9083ab04bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 21:52:38.164644 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:38.164609 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-m2sxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 21:52:38.165054 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:38.164678 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" podUID="c73ef128-5710-4654-a155-4b9083ab04bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 21:52:39.164227 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:39.164188 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-m2sxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 21:52:39.164414 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:39.164244 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" podUID="c73ef128-5710-4654-a155-4b9083ab04bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 21:52:40.164898 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:40.164861 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-m2sxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 21:52:40.165436 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:40.164956 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" podUID="c73ef128-5710-4654-a155-4b9083ab04bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 21:52:41.165048 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:41.165009 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-m2sxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 21:52:41.165510 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:41.165100 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" podUID="c73ef128-5710-4654-a155-4b9083ab04bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 21:52:42.164948 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:42.164915 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-m2sxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 21:52:42.165139 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:42.165001 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" podUID="c73ef128-5710-4654-a155-4b9083ab04bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 21:52:43.164886 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:43.164852 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-m2sxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 21:52:43.165088 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:43.164940 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" podUID="c73ef128-5710-4654-a155-4b9083ab04bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 21:52:43.706000 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:43.705932 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" podStartSLOduration=8.705911769 podStartE2EDuration="8.705911769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:37.176302955 +0000 UTC m=+2332.731002731" watchObservedRunningTime="2026-10-30 21:52:43.705911769 +0000 UTC m=+2339.260611546" Oct 30 21:52:43.731064 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:43.727566 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:52:44.164857 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:44.164817 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-m2sxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 21:52:44.165031 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:44.164888 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" podUID="c73ef128-5710-4654-a155-4b9083ab04bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 21:52:46.202147 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:46.202105 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-m2sxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:52:46.202667 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:46.202636 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" podUID="c73ef128-5710-4654-a155-4b9083ab04bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:52:46.321494 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:46.321463 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" Oct 30 21:52:46.321960 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:46.321932 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" Oct 30 21:52:46.322855 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:46.322835 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" Oct 30 21:52:46.496933 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:46.496904 4560 scope.go:122] "RemoveContainer" containerID="13a15df29b897c7e583c5c975d5d009fb980c5a2671964587df825a2bae247f7" Oct 30 21:52:46.507206 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:46.507185 4560 scope.go:122] "RemoveContainer" containerID="35ab5f78cd38aaa19f25027fa74edafef9281ecc96f8447bd39f4d49ed9601d5" Oct 30 21:52:46.516560 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:46.516541 4560 scope.go:122] "RemoveContainer" containerID="e485c5a5bf9cc0a7aa4c1fd2652af6f3e110ec2bd56fc4cd68feffce114935c1" Oct 30 21:52:46.525032 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:46.525005 4560 scope.go:122] "RemoveContainer" containerID="d5577bf78c3f4b709df4f71e206343cd942be78abf666c9d116d9f4df3cae41d" Oct 30 21:52:46.563149 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:46.563018 4560 scope.go:122] "RemoveContainer" containerID="078dc5a1eb234899b375487b465286c600b4928df9e608954ecc29067dc06c2e" Oct 30 21:52:46.598512 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:46.598488 4560 scope.go:122] "RemoveContainer" containerID="e913f3bcdba072fd1d0bc6f9dfbabddbe8ae0346cc6c35ee68b41d4888ea81e6" Oct 30 21:52:46.607691 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:46.607669 4560 scope.go:122] "RemoveContainer" containerID="91cd6fb03776777d6566ef01ab27150deb955f2b05992d6714b630f8a0cc44b3" Oct 30 21:52:46.621531 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:46.618827 4560 scope.go:122] "RemoveContainer" containerID="34c6762cd084909ae52bccaf07e4872089b260b9072ac34ed0b9f9634e12afc4" Oct 30 21:52:46.703397 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:46.701579 4560 scope.go:122] "RemoveContainer" containerID="7815a81d8ba36e1456d4cfc054ca7353c44a8041754bceb5dc58ca97ebb0ceaf" Oct 30 21:52:49.684923 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:52:49.684882 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:52:51.391705 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:51.391670 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc"] Oct 30 21:52:51.392257 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:51.391967 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" podUID="16e40614-67ae-43f6-b284-6f09fbc31b08" containerName="istio-proxy" containerID="cri-o://96498f510a6c123d37b3f0d531bfe6eca88e78257462d40596850ac567f50104" gracePeriod=30 Oct 30 21:52:56.653792 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:56.653735 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" Oct 30 21:52:56.710578 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:56.710549 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16e40614-67ae-43f6-b284-6f09fbc31b08-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16e40614-67ae-43f6-b284-6f09fbc31b08-workload-certs\") pod \"16e40614-67ae-43f6-b284-6f09fbc31b08\" (UID: \"16e40614-67ae-43f6-b284-6f09fbc31b08\") " Oct 30 21:52:56.710744 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:56.710588 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16e40614-67ae-43f6-b284-6f09fbc31b08-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16e40614-67ae-43f6-b284-6f09fbc31b08-istio-envoy\") pod \"16e40614-67ae-43f6-b284-6f09fbc31b08\" (UID: \"16e40614-67ae-43f6-b284-6f09fbc31b08\") " Oct 30 21:52:56.710744 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:56.710618 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16e40614-67ae-43f6-b284-6f09fbc31b08-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16e40614-67ae-43f6-b284-6f09fbc31b08-istiod-ca-cert\") pod \"16e40614-67ae-43f6-b284-6f09fbc31b08\" (UID: \"16e40614-67ae-43f6-b284-6f09fbc31b08\") " Oct 30 21:52:56.710744 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:56.710663 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16e40614-67ae-43f6-b284-6f09fbc31b08-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e40614-67ae-43f6-b284-6f09fbc31b08-credential-socket\") pod \"16e40614-67ae-43f6-b284-6f09fbc31b08\" (UID: \"16e40614-67ae-43f6-b284-6f09fbc31b08\") " Oct 30 21:52:56.710913 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:56.710809 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16e40614-67ae-43f6-b284-6f09fbc31b08-workload-certs" pod "16e40614-67ae-43f6-b284-6f09fbc31b08" (UID: "16e40614-67ae-43f6-b284-6f09fbc31b08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:56.710913 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:56.710846 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16e40614-67ae-43f6-b284-6f09fbc31b08-kube-api-access-q8d5z\" (UniqueName: \"kubernetes.io/projected/16e40614-67ae-43f6-b284-6f09fbc31b08-kube-api-access-q8d5z\") pod \"16e40614-67ae-43f6-b284-6f09fbc31b08\" (UID: \"16e40614-67ae-43f6-b284-6f09fbc31b08\") " Oct 30 21:52:56.711025 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:56.710918 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16e40614-67ae-43f6-b284-6f09fbc31b08-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16e40614-67ae-43f6-b284-6f09fbc31b08-istio-podinfo\") pod \"16e40614-67ae-43f6-b284-6f09fbc31b08\" (UID: \"16e40614-67ae-43f6-b284-6f09fbc31b08\") " Oct 30 21:52:56.711025 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:56.710961 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16e40614-67ae-43f6-b284-6f09fbc31b08-istio-token\" (UniqueName: \"kubernetes.io/projected/16e40614-67ae-43f6-b284-6f09fbc31b08-istio-token\") pod \"16e40614-67ae-43f6-b284-6f09fbc31b08\" (UID: \"16e40614-67ae-43f6-b284-6f09fbc31b08\") " Oct 30 21:52:56.711025 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:56.710997 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16e40614-67ae-43f6-b284-6f09fbc31b08-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16e40614-67ae-43f6-b284-6f09fbc31b08-istio-data\") pod \"16e40614-67ae-43f6-b284-6f09fbc31b08\" (UID: \"16e40614-67ae-43f6-b284-6f09fbc31b08\") " Oct 30 21:52:56.711025 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:56.710957 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16e40614-67ae-43f6-b284-6f09fbc31b08-credential-socket" pod "16e40614-67ae-43f6-b284-6f09fbc31b08" (UID: "16e40614-67ae-43f6-b284-6f09fbc31b08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:56.711025 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:56.711005 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e40614-67ae-43f6-b284-6f09fbc31b08-istiod-ca-cert" pod "16e40614-67ae-43f6-b284-6f09fbc31b08" (UID: "16e40614-67ae-43f6-b284-6f09fbc31b08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:56.711288 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:56.711038 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16e40614-67ae-43f6-b284-6f09fbc31b08-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e40614-67ae-43f6-b284-6f09fbc31b08-workload-socket\") pod \"16e40614-67ae-43f6-b284-6f09fbc31b08\" (UID: \"16e40614-67ae-43f6-b284-6f09fbc31b08\") " Oct 30 21:52:56.711372 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:56.711320 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16e40614-67ae-43f6-b284-6f09fbc31b08-istio-data" pod "16e40614-67ae-43f6-b284-6f09fbc31b08" (UID: "16e40614-67ae-43f6-b284-6f09fbc31b08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:56.711431 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:56.711408 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16e40614-67ae-43f6-b284-6f09fbc31b08-workload-socket" pod "16e40614-67ae-43f6-b284-6f09fbc31b08" (UID: "16e40614-67ae-43f6-b284-6f09fbc31b08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:56.711833 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:56.711806 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16e40614-67ae-43f6-b284-6f09fbc31b08-istio-data\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.711929 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:56.711836 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e40614-67ae-43f6-b284-6f09fbc31b08-workload-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.711929 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:56.711854 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16e40614-67ae-43f6-b284-6f09fbc31b08-workload-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.711929 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:56.711869 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16e40614-67ae-43f6-b284-6f09fbc31b08-istiod-ca-cert\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.711929 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:56.711884 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e40614-67ae-43f6-b284-6f09fbc31b08-credential-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.713012 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:56.712986 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16e40614-67ae-43f6-b284-6f09fbc31b08-istio-podinfo" pod "16e40614-67ae-43f6-b284-6f09fbc31b08" (UID: "16e40614-67ae-43f6-b284-6f09fbc31b08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:52:56.713099 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:56.713070 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e40614-67ae-43f6-b284-6f09fbc31b08-istio-token" pod "16e40614-67ae-43f6-b284-6f09fbc31b08" (UID: "16e40614-67ae-43f6-b284-6f09fbc31b08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:56.713186 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:56.713167 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16e40614-67ae-43f6-b284-6f09fbc31b08-istio-envoy" pod "16e40614-67ae-43f6-b284-6f09fbc31b08" (UID: "16e40614-67ae-43f6-b284-6f09fbc31b08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:56.713222 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:56.713198 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e40614-67ae-43f6-b284-6f09fbc31b08-kube-api-access-q8d5z" pod "16e40614-67ae-43f6-b284-6f09fbc31b08" (UID: "16e40614-67ae-43f6-b284-6f09fbc31b08"). InnerVolumeSpecName "kube-api-access-q8d5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:56.813078 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:56.813043 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16e40614-67ae-43f6-b284-6f09fbc31b08-istio-podinfo\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.813078 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:56.813073 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16e40614-67ae-43f6-b284-6f09fbc31b08-istio-token\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.813265 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:56.813086 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16e40614-67ae-43f6-b284-6f09fbc31b08-istio-envoy\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:56.813265 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:56.813100 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8d5z\" (UniqueName: \"kubernetes.io/projected/16e40614-67ae-43f6-b284-6f09fbc31b08-kube-api-access-q8d5z\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:57.243940 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:57.243904 4560 generic.go:356] "Generic (PLEG): container finished" podID="16e40614-67ae-43f6-b284-6f09fbc31b08" containerID="96498f510a6c123d37b3f0d531bfe6eca88e78257462d40596850ac567f50104" exitCode=0 Oct 30 21:52:57.244107 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:57.243951 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" Oct 30 21:52:57.244107 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:57.243961 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" event={"ID":"16e40614-67ae-43f6-b284-6f09fbc31b08","Type":"ContainerDied","Data":"96498f510a6c123d37b3f0d531bfe6eca88e78257462d40596850ac567f50104"} Oct 30 21:52:57.244107 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:57.244022 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc" event={"ID":"16e40614-67ae-43f6-b284-6f09fbc31b08","Type":"ContainerDied","Data":"264ea75e6f7429cb87af50c80f5543abc787abbfdd3982c0c9e685bfeb9ba52d"} Oct 30 21:52:57.244107 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:57.244050 4560 scope.go:122] "RemoveContainer" containerID="96498f510a6c123d37b3f0d531bfe6eca88e78257462d40596850ac567f50104" Oct 30 21:52:57.254441 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:57.254418 4560 scope.go:122] "RemoveContainer" containerID="96498f510a6c123d37b3f0d531bfe6eca88e78257462d40596850ac567f50104" Oct 30 21:52:57.254693 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:52:57.254675 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96498f510a6c123d37b3f0d531bfe6eca88e78257462d40596850ac567f50104\": container with ID starting with 96498f510a6c123d37b3f0d531bfe6eca88e78257462d40596850ac567f50104 not found: ID does not exist" containerID="96498f510a6c123d37b3f0d531bfe6eca88e78257462d40596850ac567f50104" Oct 30 21:52:57.254749 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:57.254705 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96498f510a6c123d37b3f0d531bfe6eca88e78257462d40596850ac567f50104"} err="failed to get container status \"96498f510a6c123d37b3f0d531bfe6eca88e78257462d40596850ac567f50104\": rpc error: code = NotFound desc = could not find container \"96498f510a6c123d37b3f0d531bfe6eca88e78257462d40596850ac567f50104\": container with ID starting with 96498f510a6c123d37b3f0d531bfe6eca88e78257462d40596850ac567f50104 not found: ID does not exist" Oct 30 21:52:57.401467 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:57.401441 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc"] Oct 30 21:52:57.448595 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:57.448569 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-dq4sc"] Oct 30 21:52:58.576455 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:52:58.576423 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16e40614-67ae-43f6-b284-6f09fbc31b08" path="/var/lib/kubelet/pods/16e40614-67ae-43f6-b284-6f09fbc31b08/volumes" Oct 30 21:53:19.685495 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:53:19.685460 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:53:46.936369 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:53:46.936321 4560 scope.go:122] "RemoveContainer" containerID="47fbdc66fda9184e7943037954c278bbea0fbd605323402ffd363851d46d5893" Oct 30 21:53:46.945864 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:53:46.945841 4560 scope.go:122] "RemoveContainer" containerID="e3f3ccd8183b96ed390b8205ed335dbdf53d896cb087e6b923b809efa46b4855" Oct 30 21:53:49.684905 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:53:49.684867 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:53:58.519495 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:53:58.519459 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:54:19.713235 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:54:19.713197 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:54:22.398377 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:22.398333 4560 generic.go:356] "Generic (PLEG): container finished" podID="7a8c5ad5-8b50-4795-9596-d951de399015" containerID="ddf18f19c908cd1807a1db3b7a9a0ff05d35e44448bf0556ee0ea9d41d3c2276" exitCode=0 Oct 30 21:54:22.398796 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:22.398407 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" event={"ID":"7a8c5ad5-8b50-4795-9596-d951de399015","Type":"ContainerDied","Data":"ddf18f19c908cd1807a1db3b7a9a0ff05d35e44448bf0556ee0ea9d41d3c2276"} Oct 30 21:54:22.398857 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:22.398840 4560 scope.go:122] "RemoveContainer" containerID="ddf18f19c908cd1807a1db3b7a9a0ff05d35e44448bf0556ee0ea9d41d3c2276" Oct 30 21:54:22.400367 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:22.400321 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:54:22.400439 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:22.400403 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:54:22.400475 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:22.400437 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:54:23.406111 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:23.406078 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" event={"ID":"7a8c5ad5-8b50-4795-9596-d951de399015","Type":"ContainerStarted","Data":"fe5f65e4f226f1b28a6f9e2b05ee99a5d20b3c8f64cef8c788b2aa513711e589"} Oct 30 21:54:24.052554 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:24.052517 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" Oct 30 21:54:24.052865 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:24.052836 4560 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 21:54:24.053024 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:24.052894 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" podUID="7a8c5ad5-8b50-4795-9596-d951de399015" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 21:54:25.051981 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:25.051942 4560 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 21:54:25.052540 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:25.052006 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" podUID="7a8c5ad5-8b50-4795-9596-d951de399015" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 21:54:26.053112 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:26.053069 4560 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 21:54:26.053557 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:26.053152 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" podUID="7a8c5ad5-8b50-4795-9596-d951de399015" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 21:54:27.052492 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:27.052450 4560 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 21:54:27.052673 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:27.052504 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" podUID="7a8c5ad5-8b50-4795-9596-d951de399015" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 21:54:28.052800 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:28.052760 4560 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 21:54:28.053196 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:28.052843 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" podUID="7a8c5ad5-8b50-4795-9596-d951de399015" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 21:54:29.311108 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:29.311075 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" Oct 30 21:54:29.311691 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:29.311670 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" Oct 30 21:54:29.312445 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:29.312425 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" Oct 30 21:54:36.142280 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:36.142235 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd"] Oct 30 21:54:36.142742 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:36.142460 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" podUID="bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b" containerName="istio-proxy" containerID="cri-o://2b0db5528a66e23021aee1e96044a644cb5fe268863b6ba26c58c267afb18bb4" gracePeriod=30 Oct 30 21:54:36.766043 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:36.766008 4560 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:54:36.766223 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:36.766082 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" podUID="bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:54:37.260488 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:37.260452 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh"] Oct 30 21:54:37.260949 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:37.260759 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" podUID="1bfd4837-aa59-491c-8e32-770bf1451706" containerName="istio-proxy" containerID="cri-o://3ab8eb4bd46329cad45ed1a5fd4f76925d0a8e00415497fa1667253f0ec8e7f9" gracePeriod=30 Oct 30 21:54:37.552204 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:37.552131 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8"] Oct 30 21:54:37.552379 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:37.552324 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" podUID="5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4" containerName="istio-proxy" containerID="cri-o://303d0c48ce35ea81add41aed6efd79a2c281718a209cea2d4e5646bbbfd98264" gracePeriod=30 Oct 30 21:54:41.406107 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:41.406048 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" Oct 30 21:54:41.475866 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:41.475840 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-istio-envoy\") pod \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\" (UID: \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\") " Oct 30 21:54:41.475978 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:41.475877 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-istiod-ca-cert\") pod \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\" (UID: \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\") " Oct 30 21:54:41.475978 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:41.475916 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-istio-data\") pod \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\" (UID: \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\") " Oct 30 21:54:41.475978 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:41.475967 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-workload-certs\") pod \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\" (UID: \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\") " Oct 30 21:54:41.476146 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:41.476002 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-istio-token\" (UniqueName: \"kubernetes.io/projected/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-istio-token\") pod \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\" (UID: \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\") " Oct 30 21:54:41.476146 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:41.476054 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-credential-socket\") pod \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\" (UID: \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\") " Oct 30 21:54:41.476146 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:41.476114 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-workload-socket\") pod \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\" (UID: \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\") " Oct 30 21:54:41.476296 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:41.476154 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-istio-podinfo\") pod \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\" (UID: \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\") " Oct 30 21:54:41.476296 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:41.476196 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-kube-api-access-98wf9\" (UniqueName: \"kubernetes.io/projected/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-kube-api-access-98wf9\") pod \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\" (UID: \"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b\") " Oct 30 21:54:41.476296 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:41.476247 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-istio-data" pod "bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b" (UID: "bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:41.476471 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:41.476408 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-workload-socket" pod "bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b" (UID: "bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:41.476534 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:41.476490 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-istiod-ca-cert" pod "bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b" (UID: "bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:41.476534 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:41.476484 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-credential-socket" pod "bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b" (UID: "bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:41.476638 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:41.476546 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-workload-certs" pod "bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b" (UID: "bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:41.476874 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:41.476857 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-istiod-ca-cert\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:41.476947 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:41.476878 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-istio-data\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:41.476947 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:41.476893 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-workload-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:41.476947 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:41.476906 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-credential-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:41.476947 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:41.476919 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-workload-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:41.478411 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:41.478370 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-istio-podinfo" pod "bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b" (UID: "bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:54:41.478700 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:41.478679 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-istio-token" pod "bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b" (UID: "bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:41.478785 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:41.478713 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-istio-envoy" pod "bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b" (UID: "bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:41.478785 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:41.478717 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-kube-api-access-98wf9" pod "bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b" (UID: "bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b"). InnerVolumeSpecName "kube-api-access-98wf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:41.586896 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:41.580022 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-istio-envoy\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:41.586896 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:41.580054 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-istio-token\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:41.586896 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:41.580071 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-istio-podinfo\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:41.586896 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:41.580086 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98wf9\" (UniqueName: \"kubernetes.io/projected/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b-kube-api-access-98wf9\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:41.594098 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:41.594069 4560 generic.go:356] "Generic (PLEG): container finished" podID="bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b" containerID="2b0db5528a66e23021aee1e96044a644cb5fe268863b6ba26c58c267afb18bb4" exitCode=0 Oct 30 21:54:41.594235 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:41.594119 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" event={"ID":"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b","Type":"ContainerDied","Data":"2b0db5528a66e23021aee1e96044a644cb5fe268863b6ba26c58c267afb18bb4"} Oct 30 21:54:41.594235 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:41.594131 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" Oct 30 21:54:41.594235 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:41.594194 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd" event={"ID":"bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b","Type":"ContainerDied","Data":"8358ed816053f4d4d2415495dc623126bb24660ee342b95846d48e1417008f29"} Oct 30 21:54:41.594235 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:41.594217 4560 scope.go:122] "RemoveContainer" containerID="2b0db5528a66e23021aee1e96044a644cb5fe268863b6ba26c58c267afb18bb4" Oct 30 21:54:41.622967 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:41.622938 4560 scope.go:122] "RemoveContainer" containerID="2b0db5528a66e23021aee1e96044a644cb5fe268863b6ba26c58c267afb18bb4" Oct 30 21:54:41.625552 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:54:41.625486 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b0db5528a66e23021aee1e96044a644cb5fe268863b6ba26c58c267afb18bb4\": container with ID starting with 2b0db5528a66e23021aee1e96044a644cb5fe268863b6ba26c58c267afb18bb4 not found: ID does not exist" containerID="2b0db5528a66e23021aee1e96044a644cb5fe268863b6ba26c58c267afb18bb4" Oct 30 21:54:41.625552 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:41.625532 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b0db5528a66e23021aee1e96044a644cb5fe268863b6ba26c58c267afb18bb4"} err="failed to get container status \"2b0db5528a66e23021aee1e96044a644cb5fe268863b6ba26c58c267afb18bb4\": rpc error: code = NotFound desc = could not find container \"2b0db5528a66e23021aee1e96044a644cb5fe268863b6ba26c58c267afb18bb4\": container with ID starting with 2b0db5528a66e23021aee1e96044a644cb5fe268863b6ba26c58c267afb18bb4 not found: ID does not exist" Oct 30 21:54:41.755674 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:41.755643 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd"] Oct 30 21:54:41.785591 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:41.785561 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-qntfd"] Oct 30 21:54:41.855943 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:41.855909 4560 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:54:41.856095 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:41.855965 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" podUID="5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:54:42.517592 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:42.517567 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" Oct 30 21:54:42.577928 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:42.577903 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b" path="/var/lib/kubelet/pods/bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b/volumes" Oct 30 21:54:42.595221 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:42.595199 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bfd4837-aa59-491c-8e32-770bf1451706-istio-token\" (UniqueName: \"kubernetes.io/projected/1bfd4837-aa59-491c-8e32-770bf1451706-istio-token\") pod \"1bfd4837-aa59-491c-8e32-770bf1451706\" (UID: \"1bfd4837-aa59-491c-8e32-770bf1451706\") " Oct 30 21:54:42.595355 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:42.595255 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1bfd4837-aa59-491c-8e32-770bf1451706-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bfd4837-aa59-491c-8e32-770bf1451706-istio-podinfo\") pod \"1bfd4837-aa59-491c-8e32-770bf1451706\" (UID: \"1bfd4837-aa59-491c-8e32-770bf1451706\") " Oct 30 21:54:42.595430 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:42.595417 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bfd4837-aa59-491c-8e32-770bf1451706-kube-api-access-bxgrn\" (UniqueName: \"kubernetes.io/projected/1bfd4837-aa59-491c-8e32-770bf1451706-kube-api-access-bxgrn\") pod \"1bfd4837-aa59-491c-8e32-770bf1451706\" (UID: \"1bfd4837-aa59-491c-8e32-770bf1451706\") " Oct 30 21:54:42.595498 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:42.595484 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bfd4837-aa59-491c-8e32-770bf1451706-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bfd4837-aa59-491c-8e32-770bf1451706-istio-data\") pod \"1bfd4837-aa59-491c-8e32-770bf1451706\" (UID: \"1bfd4837-aa59-491c-8e32-770bf1451706\") " Oct 30 21:54:42.595557 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:42.595525 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bfd4837-aa59-491c-8e32-770bf1451706-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfd4837-aa59-491c-8e32-770bf1451706-workload-socket\") pod \"1bfd4837-aa59-491c-8e32-770bf1451706\" (UID: \"1bfd4837-aa59-491c-8e32-770bf1451706\") " Oct 30 21:54:42.595619 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:42.595560 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bfd4837-aa59-491c-8e32-770bf1451706-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bfd4837-aa59-491c-8e32-770bf1451706-workload-certs\") pod \"1bfd4837-aa59-491c-8e32-770bf1451706\" (UID: \"1bfd4837-aa59-491c-8e32-770bf1451706\") " Oct 30 21:54:42.595619 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:42.595601 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bfd4837-aa59-491c-8e32-770bf1451706-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bfd4837-aa59-491c-8e32-770bf1451706-istiod-ca-cert\") pod \"1bfd4837-aa59-491c-8e32-770bf1451706\" (UID: \"1bfd4837-aa59-491c-8e32-770bf1451706\") " Oct 30 21:54:42.595722 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:42.595646 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bfd4837-aa59-491c-8e32-770bf1451706-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bfd4837-aa59-491c-8e32-770bf1451706-istio-envoy\") pod \"1bfd4837-aa59-491c-8e32-770bf1451706\" (UID: \"1bfd4837-aa59-491c-8e32-770bf1451706\") " Oct 30 21:54:42.595722 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:42.595713 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bfd4837-aa59-491c-8e32-770bf1451706-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfd4837-aa59-491c-8e32-770bf1451706-credential-socket\") pod \"1bfd4837-aa59-491c-8e32-770bf1451706\" (UID: \"1bfd4837-aa59-491c-8e32-770bf1451706\") " Oct 30 21:54:42.595833 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:42.595740 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfd4837-aa59-491c-8e32-770bf1451706-istio-data" pod "1bfd4837-aa59-491c-8e32-770bf1451706" (UID: "1bfd4837-aa59-491c-8e32-770bf1451706"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:42.595833 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:42.595772 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfd4837-aa59-491c-8e32-770bf1451706-workload-socket" pod "1bfd4837-aa59-491c-8e32-770bf1451706" (UID: "1bfd4837-aa59-491c-8e32-770bf1451706"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:42.596046 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:42.596025 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfd4837-aa59-491c-8e32-770bf1451706-credential-socket" pod "1bfd4837-aa59-491c-8e32-770bf1451706" (UID: "1bfd4837-aa59-491c-8e32-770bf1451706"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:42.596146 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:42.596106 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bfd4837-aa59-491c-8e32-770bf1451706-istiod-ca-cert" pod "1bfd4837-aa59-491c-8e32-770bf1451706" (UID: "1bfd4837-aa59-491c-8e32-770bf1451706"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:42.596734 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:42.596327 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfd4837-aa59-491c-8e32-770bf1451706-credential-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:42.596734 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:42.596368 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bfd4837-aa59-491c-8e32-770bf1451706-istio-data\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:42.596734 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:42.596384 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfd4837-aa59-491c-8e32-770bf1451706-workload-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:42.596734 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:42.596398 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bfd4837-aa59-491c-8e32-770bf1451706-istiod-ca-cert\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:42.596734 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:42.596635 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfd4837-aa59-491c-8e32-770bf1451706-workload-certs" pod "1bfd4837-aa59-491c-8e32-770bf1451706" (UID: "1bfd4837-aa59-491c-8e32-770bf1451706"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:42.597900 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:42.597879 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bfd4837-aa59-491c-8e32-770bf1451706-kube-api-access-bxgrn" pod "1bfd4837-aa59-491c-8e32-770bf1451706" (UID: "1bfd4837-aa59-491c-8e32-770bf1451706"). InnerVolumeSpecName "kube-api-access-bxgrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:42.597994 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:42.597969 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bfd4837-aa59-491c-8e32-770bf1451706-istio-token" pod "1bfd4837-aa59-491c-8e32-770bf1451706" (UID: "1bfd4837-aa59-491c-8e32-770bf1451706"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:42.597994 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:42.597976 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfd4837-aa59-491c-8e32-770bf1451706-istio-envoy" pod "1bfd4837-aa59-491c-8e32-770bf1451706" (UID: "1bfd4837-aa59-491c-8e32-770bf1451706"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:42.598067 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:42.598057 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1bfd4837-aa59-491c-8e32-770bf1451706-istio-podinfo" pod "1bfd4837-aa59-491c-8e32-770bf1451706" (UID: "1bfd4837-aa59-491c-8e32-770bf1451706"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:54:42.602984 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:42.602961 4560 generic.go:356] "Generic (PLEG): container finished" podID="1bfd4837-aa59-491c-8e32-770bf1451706" containerID="3ab8eb4bd46329cad45ed1a5fd4f76925d0a8e00415497fa1667253f0ec8e7f9" exitCode=0 Oct 30 21:54:42.603085 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:42.603073 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" Oct 30 21:54:42.623597 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:42.623575 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" event={"ID":"1bfd4837-aa59-491c-8e32-770bf1451706","Type":"ContainerDied","Data":"3ab8eb4bd46329cad45ed1a5fd4f76925d0a8e00415497fa1667253f0ec8e7f9"} Oct 30 21:54:42.623690 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:42.623626 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh" event={"ID":"1bfd4837-aa59-491c-8e32-770bf1451706","Type":"ContainerDied","Data":"0c31d18d802f93990cee3c89a94222737bba0499be114dff11e325fe6227add4"} Oct 30 21:54:42.623690 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:42.623650 4560 scope.go:122] "RemoveContainer" containerID="3ab8eb4bd46329cad45ed1a5fd4f76925d0a8e00415497fa1667253f0ec8e7f9" Oct 30 21:54:42.698906 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:42.698867 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bfd4837-aa59-491c-8e32-770bf1451706-istio-envoy\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:42.699125 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:42.698965 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bfd4837-aa59-491c-8e32-770bf1451706-istio-token\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:42.699125 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:42.698985 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bfd4837-aa59-491c-8e32-770bf1451706-istio-podinfo\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:42.699125 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:42.699002 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxgrn\" (UniqueName: \"kubernetes.io/projected/1bfd4837-aa59-491c-8e32-770bf1451706-kube-api-access-bxgrn\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:42.699125 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:42.699018 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bfd4837-aa59-491c-8e32-770bf1451706-workload-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:42.776452 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:42.776417 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh"] Oct 30 21:54:42.779225 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:42.779197 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-19dd4f-openshift-default-69f45f44bc-2n5hh"] Oct 30 21:54:42.789410 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:42.789386 4560 scope.go:122] "RemoveContainer" containerID="3ab8eb4bd46329cad45ed1a5fd4f76925d0a8e00415497fa1667253f0ec8e7f9" Oct 30 21:54:42.789773 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:54:42.789747 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ab8eb4bd46329cad45ed1a5fd4f76925d0a8e00415497fa1667253f0ec8e7f9\": container with ID starting with 3ab8eb4bd46329cad45ed1a5fd4f76925d0a8e00415497fa1667253f0ec8e7f9 not found: ID does not exist" containerID="3ab8eb4bd46329cad45ed1a5fd4f76925d0a8e00415497fa1667253f0ec8e7f9" Oct 30 21:54:42.789875 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:42.789787 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ab8eb4bd46329cad45ed1a5fd4f76925d0a8e00415497fa1667253f0ec8e7f9"} err="failed to get container status \"3ab8eb4bd46329cad45ed1a5fd4f76925d0a8e00415497fa1667253f0ec8e7f9\": rpc error: code = NotFound desc = could not find container \"3ab8eb4bd46329cad45ed1a5fd4f76925d0a8e00415497fa1667253f0ec8e7f9\": container with ID starting with 3ab8eb4bd46329cad45ed1a5fd4f76925d0a8e00415497fa1667253f0ec8e7f9 not found: ID does not exist" Oct 30 21:54:42.955697 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:42.955666 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" Oct 30 21:54:43.105060 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:43.104984 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-istio-token\" (UniqueName: \"kubernetes.io/projected/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-istio-token\") pod \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\" (UID: \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\") " Oct 30 21:54:43.105060 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:43.105029 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-workload-socket\") pod \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\" (UID: \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\") " Oct 30 21:54:43.105297 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:43.105063 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-istio-podinfo\") pod \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\" (UID: \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\") " Oct 30 21:54:43.105297 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:43.105136 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-kube-api-access-hkcgd\" (UniqueName: \"kubernetes.io/projected/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-kube-api-access-hkcgd\") pod \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\" (UID: \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\") " Oct 30 21:54:43.105297 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:43.105159 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-istio-envoy\") pod \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\" (UID: \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\") " Oct 30 21:54:43.105297 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:43.105186 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-istio-data\") pod \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\" (UID: \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\") " Oct 30 21:54:43.105297 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:43.105223 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-credential-socket\") pod \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\" (UID: \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\") " Oct 30 21:54:43.105297 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:43.105284 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-istiod-ca-cert\") pod \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\" (UID: \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\") " Oct 30 21:54:43.105625 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:43.105317 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-workload-socket" pod "5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4" (UID: "5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:43.105625 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:43.105367 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-workload-certs\") pod \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\" (UID: \"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4\") " Oct 30 21:54:43.105625 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:43.105560 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-istio-data" pod "5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4" (UID: "5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:43.105789 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:43.105650 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-credential-socket" pod "5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4" (UID: "5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:43.105789 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:43.105775 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-workload-certs" pod "5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4" (UID: "5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:43.105885 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:43.105849 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-istiod-ca-cert" pod "5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4" (UID: "5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:43.106038 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:43.106016 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-workload-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:43.106109 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:43.106043 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-istio-data\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:43.106109 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:43.106061 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-credential-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:43.106109 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:43.106078 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-istiod-ca-cert\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:43.106109 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:43.106091 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-workload-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:43.107379 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:43.107355 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-istio-podinfo" pod "5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4" (UID: "5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:54:43.108014 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:43.107993 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-istio-envoy" pod "5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4" (UID: "5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:43.108014 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:43.108004 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-istio-token" pod "5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4" (UID: "5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:43.108133 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:43.108119 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-kube-api-access-hkcgd" pod "5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4" (UID: "5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4"). InnerVolumeSpecName "kube-api-access-hkcgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:43.207212 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:43.207171 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-istio-token\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:43.207212 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:43.207203 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-istio-podinfo\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:43.207212 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:43.207220 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkcgd\" (UniqueName: \"kubernetes.io/projected/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-kube-api-access-hkcgd\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:43.207475 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:43.207232 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4-istio-envoy\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:43.611351 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:43.611299 4560 generic.go:356] "Generic (PLEG): container finished" podID="5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4" containerID="303d0c48ce35ea81add41aed6efd79a2c281718a209cea2d4e5646bbbfd98264" exitCode=0 Oct 30 21:54:43.611351 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:43.611351 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" Oct 30 21:54:43.611876 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:43.611350 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" event={"ID":"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4","Type":"ContainerDied","Data":"303d0c48ce35ea81add41aed6efd79a2c281718a209cea2d4e5646bbbfd98264"} Oct 30 21:54:43.611876 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:43.611409 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8" event={"ID":"5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4","Type":"ContainerDied","Data":"21d32539cd13fce101fb82f51a83d71e1f9f3a03e035cd3d8261e59d8e907181"} Oct 30 21:54:43.611876 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:43.611440 4560 scope.go:122] "RemoveContainer" containerID="303d0c48ce35ea81add41aed6efd79a2c281718a209cea2d4e5646bbbfd98264" Oct 30 21:54:43.628543 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:43.628524 4560 scope.go:122] "RemoveContainer" containerID="303d0c48ce35ea81add41aed6efd79a2c281718a209cea2d4e5646bbbfd98264" Oct 30 21:54:43.628833 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:54:43.628811 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"303d0c48ce35ea81add41aed6efd79a2c281718a209cea2d4e5646bbbfd98264\": container with ID starting with 303d0c48ce35ea81add41aed6efd79a2c281718a209cea2d4e5646bbbfd98264 not found: ID does not exist" containerID="303d0c48ce35ea81add41aed6efd79a2c281718a209cea2d4e5646bbbfd98264" Oct 30 21:54:43.628932 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:43.628838 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"303d0c48ce35ea81add41aed6efd79a2c281718a209cea2d4e5646bbbfd98264"} err="failed to get container status \"303d0c48ce35ea81add41aed6efd79a2c281718a209cea2d4e5646bbbfd98264\": rpc error: code = NotFound desc = could not find container \"303d0c48ce35ea81add41aed6efd79a2c281718a209cea2d4e5646bbbfd98264\": container with ID starting with 303d0c48ce35ea81add41aed6efd79a2c281718a209cea2d4e5646bbbfd98264 not found: ID does not exist" Oct 30 21:54:43.770733 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:43.770698 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8"] Oct 30 21:54:43.817506 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:43.817470 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-t9mm8"] Oct 30 21:54:44.577250 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:44.577220 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bfd4837-aa59-491c-8e32-770bf1451706" path="/var/lib/kubelet/pods/1bfd4837-aa59-491c-8e32-770bf1451706/volumes" Oct 30 21:54:44.577728 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:44.577711 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4" path="/var/lib/kubelet/pods/5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4/volumes" Oct 30 21:54:45.058351 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:45.058314 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6fc664f-k7mpl"] Oct 30 21:54:45.104903 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:45.104873 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6fc664f-k7mpl"] Oct 30 21:54:45.585660 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:45.585633 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-lln9s"] Oct 30 21:54:45.586092 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:45.586080 4560 cpu_manager.go:414] "containerMap: removing container" podUID="1bfd4837-aa59-491c-8e32-770bf1451706" containerName="istio-proxy" Oct 30 21:54:45.586151 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:45.586093 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bfd4837-aa59-491c-8e32-770bf1451706" containerName="istio-proxy" Oct 30 21:54:45.586151 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:45.586111 4560 cpu_manager.go:414] "containerMap: removing container" podUID="bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b" containerName="istio-proxy" Oct 30 21:54:45.586151 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:45.586116 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b" containerName="istio-proxy" Oct 30 21:54:45.586151 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:45.586134 4560 cpu_manager.go:414] "containerMap: removing container" podUID="5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4" containerName="istio-proxy" Oct 30 21:54:45.586151 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:45.586138 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4" containerName="istio-proxy" Oct 30 21:54:45.586151 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:45.586146 4560 cpu_manager.go:414] "containerMap: removing container" podUID="16e40614-67ae-43f6-b284-6f09fbc31b08" containerName="istio-proxy" Oct 30 21:54:45.586151 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:45.586151 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16e40614-67ae-43f6-b284-6f09fbc31b08" containerName="istio-proxy" Oct 30 21:54:45.586384 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:45.586226 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ee3ad34-eadf-4009-ac4f-bcc9ee000ce4" containerName="istio-proxy" Oct 30 21:54:45.586384 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:45.586235 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="16e40614-67ae-43f6-b284-6f09fbc31b08" containerName="istio-proxy" Oct 30 21:54:45.586384 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:45.586242 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bfd4837-aa59-491c-8e32-770bf1451706" containerName="istio-proxy" Oct 30 21:54:45.586384 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:45.586251 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="bad45380-4c5a-4ae4-9bd2-9bdb81ac2d1b" containerName="istio-proxy" Oct 30 21:54:45.590884 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:45.590868 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-lln9s" Oct 30 21:54:45.641249 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:45.641219 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9214c76a-32cc-45de-8130-a22060868346-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-bfdf30b9-lln9s\" (UID: \"9214c76a-32cc-45de-8130-a22060868346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-lln9s" Oct 30 21:54:45.641570 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:45.641541 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngj28\" (UniqueName: \"kubernetes.io/projected/9214c76a-32cc-45de-8130-a22060868346-kube-api-access-ngj28\") pod \"maas-api-revoke-keys-e2e-mt-bfdf30b9-lln9s\" (UID: \"9214c76a-32cc-45de-8130-a22060868346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-lln9s" Oct 30 21:54:45.641812 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:45.641786 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9214c76a-32cc-45de-8130-a22060868346-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-bfdf30b9-lln9s\" (UID: \"9214c76a-32cc-45de-8130-a22060868346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-lln9s" Oct 30 21:54:45.687825 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:45.687796 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:54:45.687969 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:45.687861 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:54:45.687969 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:45.687865 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:54:45.688090 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:45.688030 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:54:45.692541 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:45.692520 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-lln9s"] Oct 30 21:54:45.742968 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:45.742942 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngj28\" (UniqueName: \"kubernetes.io/projected/9214c76a-32cc-45de-8130-a22060868346-kube-api-access-ngj28\") pod \"maas-api-revoke-keys-e2e-mt-bfdf30b9-lln9s\" (UID: \"9214c76a-32cc-45de-8130-a22060868346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-lln9s" Oct 30 21:54:45.743064 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:45.743040 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9214c76a-32cc-45de-8130-a22060868346-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-bfdf30b9-lln9s\" (UID: \"9214c76a-32cc-45de-8130-a22060868346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-lln9s" Oct 30 21:54:45.743175 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:45.743161 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9214c76a-32cc-45de-8130-a22060868346-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-bfdf30b9-lln9s\" (UID: \"9214c76a-32cc-45de-8130-a22060868346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-lln9s" Oct 30 21:54:45.743672 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:45.743650 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9214c76a-32cc-45de-8130-a22060868346-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-bfdf30b9-lln9s\" (UID: \"9214c76a-32cc-45de-8130-a22060868346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-lln9s" Oct 30 21:54:45.745526 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:45.745504 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9214c76a-32cc-45de-8130-a22060868346-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-bfdf30b9-lln9s\" (UID: \"9214c76a-32cc-45de-8130-a22060868346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-lln9s" Oct 30 21:54:45.801036 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:45.801012 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngj28\" (UniqueName: \"kubernetes.io/projected/9214c76a-32cc-45de-8130-a22060868346-kube-api-access-ngj28\") pod \"maas-api-revoke-keys-e2e-mt-bfdf30b9-lln9s\" (UID: \"9214c76a-32cc-45de-8130-a22060868346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-lln9s" Oct 30 21:54:45.903628 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:45.903558 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-lln9s" Oct 30 21:54:46.291996 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:46.291969 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-lln9s"] Oct 30 21:54:46.296391 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:54:46.296331 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9214c76a_32cc_45de_8130_a22060868346.slice/crio-3752e623e9e834266a235523f57699624a8b47cb2b64adced67349ea3df9ecdd WatchSource:0}: Error finding container 3752e623e9e834266a235523f57699624a8b47cb2b64adced67349ea3df9ecdd: Status 404 returned error can't find the container with id 3752e623e9e834266a235523f57699624a8b47cb2b64adced67349ea3df9ecdd Oct 30 21:54:46.576382 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:46.576307 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e85868aa-ad8d-4c14-8c39-0514ef644922" path="/var/lib/kubelet/pods/e85868aa-ad8d-4c14-8c39-0514ef644922/volumes" Oct 30 21:54:46.652829 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:46.652804 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-lln9s" event={"ID":"9214c76a-32cc-45de-8130-a22060868346","Type":"ContainerStarted","Data":"ad3bd1addb4ef4b317eff2ced225418bb7f14f064bee5dfead2c5a43fcb60987"} Oct 30 21:54:46.652938 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:46.652871 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-lln9s" event={"ID":"9214c76a-32cc-45de-8130-a22060868346","Type":"ContainerStarted","Data":"3752e623e9e834266a235523f57699624a8b47cb2b64adced67349ea3df9ecdd"} Oct 30 21:54:46.764138 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:46.764086 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-lln9s" podStartSLOduration=1.7640733709999998 podStartE2EDuration="1.764073371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:46.762905526 +0000 UTC m=+2462.317605302" watchObservedRunningTime="2026-10-30 21:54:46.764073371 +0000 UTC m=+2462.318773200" Oct 30 21:54:46.978855 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:46.978828 4560 scope.go:122] "RemoveContainer" containerID="1cbd9e656ba2581911305c01a5a7e96ce581fe04a37cab0377361c098733a00a" Oct 30 21:54:46.989694 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:46.989675 4560 scope.go:122] "RemoveContainer" containerID="b89dc3d6be7dd622c04b91ac31852c79e76cdb88d351fd94c2b264a67cde1362" Oct 30 21:54:46.999879 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:46.999863 4560 scope.go:122] "RemoveContainer" containerID="67419d7320bc500ce260826f8212a3a007520729385f8a8ca31f4a58260a2976" Oct 30 21:54:47.660455 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:47.660418 4560 generic.go:356] "Generic (PLEG): container finished" podID="9214c76a-32cc-45de-8130-a22060868346" containerID="ad3bd1addb4ef4b317eff2ced225418bb7f14f064bee5dfead2c5a43fcb60987" exitCode=0 Oct 30 21:54:47.660854 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:47.660478 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-lln9s" event={"ID":"9214c76a-32cc-45de-8130-a22060868346","Type":"ContainerDied","Data":"ad3bd1addb4ef4b317eff2ced225418bb7f14f064bee5dfead2c5a43fcb60987"} Oct 30 21:54:48.829260 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:48.829236 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-lln9s" Oct 30 21:54:48.884971 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:48.884947 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9214c76a-32cc-45de-8130-a22060868346-kube-api-access-ngj28\" (UniqueName: \"kubernetes.io/projected/9214c76a-32cc-45de-8130-a22060868346-kube-api-access-ngj28\") pod \"9214c76a-32cc-45de-8130-a22060868346\" (UID: \"9214c76a-32cc-45de-8130-a22060868346\") " Oct 30 21:54:48.885105 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:48.884999 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9214c76a-32cc-45de-8130-a22060868346-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9214c76a-32cc-45de-8130-a22060868346-maas-api-xks-ca\") pod \"9214c76a-32cc-45de-8130-a22060868346\" (UID: \"9214c76a-32cc-45de-8130-a22060868346\") " Oct 30 21:54:48.885105 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:48.885076 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9214c76a-32cc-45de-8130-a22060868346-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9214c76a-32cc-45de-8130-a22060868346-maas-api-openshift-ca\") pod \"9214c76a-32cc-45de-8130-a22060868346\" (UID: \"9214c76a-32cc-45de-8130-a22060868346\") " Oct 30 21:54:48.885463 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:48.885438 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9214c76a-32cc-45de-8130-a22060868346-maas-api-openshift-ca" pod "9214c76a-32cc-45de-8130-a22060868346" (UID: "9214c76a-32cc-45de-8130-a22060868346"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:48.885585 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:48.885569 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9214c76a-32cc-45de-8130-a22060868346-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:48.886992 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:48.886964 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9214c76a-32cc-45de-8130-a22060868346-maas-api-xks-ca" pod "9214c76a-32cc-45de-8130-a22060868346" (UID: "9214c76a-32cc-45de-8130-a22060868346"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:48.887092 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:48.887074 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9214c76a-32cc-45de-8130-a22060868346-kube-api-access-ngj28" pod "9214c76a-32cc-45de-8130-a22060868346" (UID: "9214c76a-32cc-45de-8130-a22060868346"). InnerVolumeSpecName "kube-api-access-ngj28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:48.987239 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:48.987207 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngj28\" (UniqueName: \"kubernetes.io/projected/9214c76a-32cc-45de-8130-a22060868346-kube-api-access-ngj28\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:48.987239 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:48.987236 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9214c76a-32cc-45de-8130-a22060868346-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:49.673998 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:49.673958 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-lln9s" Oct 30 21:54:49.673998 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:49.673983 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-lln9s" event={"ID":"9214c76a-32cc-45de-8130-a22060868346","Type":"ContainerDied","Data":"3752e623e9e834266a235523f57699624a8b47cb2b64adced67349ea3df9ecdd"} Oct 30 21:54:49.674281 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:49.674027 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3752e623e9e834266a235523f57699624a8b47cb2b64adced67349ea3df9ecdd" Oct 30 21:54:49.687667 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:54:49.687629 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:54:58.205419 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:54:58.201333 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:55:19.685126 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:55:19.685082 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:55:23.269738 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:23.269692 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2"] Oct 30 21:55:23.270212 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:23.269926 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" podUID="52c80c2c-2622-43a7-ab5c-bea1f6089c90" containerName="istio-proxy" containerID="cri-o://506ec9cf0df4614b55fb5fae142b98fbc40cc9034d23ce84e022554813446fbe" gracePeriod=30 Oct 30 21:55:25.975922 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:25.975885 4560 patch_prober.go:29] interesting pod/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:55:25.976435 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:25.975946 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" podUID="52c80c2c-2622-43a7-ab5c-bea1f6089c90" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:55:28.549847 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:28.549824 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" Oct 30 21:55:28.712961 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:28.712926 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52c80c2c-2622-43a7-ab5c-bea1f6089c90-kube-api-access-p5nfc\" (UniqueName: \"kubernetes.io/projected/52c80c2c-2622-43a7-ab5c-bea1f6089c90-kube-api-access-p5nfc\") pod \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\" (UID: \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\") " Oct 30 21:55:28.712961 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:28.712967 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52c80c2c-2622-43a7-ab5c-bea1f6089c90-istio-token\" (UniqueName: \"kubernetes.io/projected/52c80c2c-2622-43a7-ab5c-bea1f6089c90-istio-token\") pod \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\" (UID: \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\") " Oct 30 21:55:28.713235 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:28.713035 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52c80c2c-2622-43a7-ab5c-bea1f6089c90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c80c2c-2622-43a7-ab5c-bea1f6089c90-istiod-ca-cert\") pod \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\" (UID: \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\") " Oct 30 21:55:28.713235 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:28.713062 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52c80c2c-2622-43a7-ab5c-bea1f6089c90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c80c2c-2622-43a7-ab5c-bea1f6089c90-workload-certs\") pod \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\" (UID: \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\") " Oct 30 21:55:28.713386 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:28.713233 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52c80c2c-2622-43a7-ab5c-bea1f6089c90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c80c2c-2622-43a7-ab5c-bea1f6089c90-credential-socket\") pod \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\" (UID: \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\") " Oct 30 21:55:28.713386 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:28.713282 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52c80c2c-2622-43a7-ab5c-bea1f6089c90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c80c2c-2622-43a7-ab5c-bea1f6089c90-istio-data\") pod \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\" (UID: \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\") " Oct 30 21:55:28.713655 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:28.713625 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52c80c2c-2622-43a7-ab5c-bea1f6089c90-istiod-ca-cert" pod "52c80c2c-2622-43a7-ab5c-bea1f6089c90" (UID: "52c80c2c-2622-43a7-ab5c-bea1f6089c90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:28.713774 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:28.713644 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c80c2c-2622-43a7-ab5c-bea1f6089c90-credential-socket" pod "52c80c2c-2622-43a7-ab5c-bea1f6089c90" (UID: "52c80c2c-2622-43a7-ab5c-bea1f6089c90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:28.713919 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:28.713897 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c80c2c-2622-43a7-ab5c-bea1f6089c90-istio-data" pod "52c80c2c-2622-43a7-ab5c-bea1f6089c90" (UID: "52c80c2c-2622-43a7-ab5c-bea1f6089c90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:28.713975 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:28.713962 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c80c2c-2622-43a7-ab5c-bea1f6089c90-workload-certs" pod "52c80c2c-2622-43a7-ab5c-bea1f6089c90" (UID: "52c80c2c-2622-43a7-ab5c-bea1f6089c90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:28.714190 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:28.714167 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52c80c2c-2622-43a7-ab5c-bea1f6089c90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c80c2c-2622-43a7-ab5c-bea1f6089c90-istio-podinfo\") pod \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\" (UID: \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\") " Oct 30 21:55:28.714266 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:28.714220 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52c80c2c-2622-43a7-ab5c-bea1f6089c90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c80c2c-2622-43a7-ab5c-bea1f6089c90-workload-socket\") pod \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\" (UID: \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\") " Oct 30 21:55:28.715056 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:28.714435 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52c80c2c-2622-43a7-ab5c-bea1f6089c90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c80c2c-2622-43a7-ab5c-bea1f6089c90-istio-envoy\") pod \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\" (UID: \"52c80c2c-2622-43a7-ab5c-bea1f6089c90\") " Oct 30 21:55:28.715056 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:28.714771 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c80c2c-2622-43a7-ab5c-bea1f6089c90-workload-socket" pod "52c80c2c-2622-43a7-ab5c-bea1f6089c90" (UID: "52c80c2c-2622-43a7-ab5c-bea1f6089c90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:28.715796 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:28.715768 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52c80c2c-2622-43a7-ab5c-bea1f6089c90-kube-api-access-p5nfc" pod "52c80c2c-2622-43a7-ab5c-bea1f6089c90" (UID: "52c80c2c-2622-43a7-ab5c-bea1f6089c90"). InnerVolumeSpecName "kube-api-access-p5nfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:28.715897 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:28.715791 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52c80c2c-2622-43a7-ab5c-bea1f6089c90-istio-token" pod "52c80c2c-2622-43a7-ab5c-bea1f6089c90" (UID: "52c80c2c-2622-43a7-ab5c-bea1f6089c90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:28.716982 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:28.716956 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c80c2c-2622-43a7-ab5c-bea1f6089c90-credential-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:55:28.717081 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:28.716985 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c80c2c-2622-43a7-ab5c-bea1f6089c90-istio-data\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:55:28.717081 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:28.717001 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c80c2c-2622-43a7-ab5c-bea1f6089c90-workload-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:55:28.717081 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:28.717016 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5nfc\" (UniqueName: \"kubernetes.io/projected/52c80c2c-2622-43a7-ab5c-bea1f6089c90-kube-api-access-p5nfc\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:55:28.717081 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:28.717030 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52c80c2c-2622-43a7-ab5c-bea1f6089c90-istio-token\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:55:28.717081 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:28.717041 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c80c2c-2622-43a7-ab5c-bea1f6089c90-istiod-ca-cert\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:55:28.717081 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:28.717053 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c80c2c-2622-43a7-ab5c-bea1f6089c90-workload-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:55:28.717661 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:28.717190 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52c80c2c-2622-43a7-ab5c-bea1f6089c90-istio-podinfo" pod "52c80c2c-2622-43a7-ab5c-bea1f6089c90" (UID: "52c80c2c-2622-43a7-ab5c-bea1f6089c90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:55:28.718398 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:28.718371 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c80c2c-2622-43a7-ab5c-bea1f6089c90-istio-envoy" pod "52c80c2c-2622-43a7-ab5c-bea1f6089c90" (UID: "52c80c2c-2622-43a7-ab5c-bea1f6089c90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:28.819086 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:28.819006 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c80c2c-2622-43a7-ab5c-bea1f6089c90-istio-podinfo\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:55:28.819086 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:28.819038 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c80c2c-2622-43a7-ab5c-bea1f6089c90-istio-envoy\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:55:29.003042 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:29.003012 4560 generic.go:356] "Generic (PLEG): container finished" podID="52c80c2c-2622-43a7-ab5c-bea1f6089c90" containerID="506ec9cf0df4614b55fb5fae142b98fbc40cc9034d23ce84e022554813446fbe" exitCode=0 Oct 30 21:55:29.003220 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:29.003058 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" Oct 30 21:55:29.003220 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:29.003097 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" event={"ID":"52c80c2c-2622-43a7-ab5c-bea1f6089c90","Type":"ContainerDied","Data":"506ec9cf0df4614b55fb5fae142b98fbc40cc9034d23ce84e022554813446fbe"} Oct 30 21:55:29.003220 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:29.003154 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2" event={"ID":"52c80c2c-2622-43a7-ab5c-bea1f6089c90","Type":"ContainerDied","Data":"3c9cc18108960889030c048b02920a15e19d9c379d217376568128a583aa80e1"} Oct 30 21:55:29.003220 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:29.003177 4560 scope.go:122] "RemoveContainer" containerID="506ec9cf0df4614b55fb5fae142b98fbc40cc9034d23ce84e022554813446fbe" Oct 30 21:55:29.013848 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:29.013831 4560 scope.go:122] "RemoveContainer" containerID="506ec9cf0df4614b55fb5fae142b98fbc40cc9034d23ce84e022554813446fbe" Oct 30 21:55:29.014121 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:55:29.014098 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"506ec9cf0df4614b55fb5fae142b98fbc40cc9034d23ce84e022554813446fbe\": container with ID starting with 506ec9cf0df4614b55fb5fae142b98fbc40cc9034d23ce84e022554813446fbe not found: ID does not exist" containerID="506ec9cf0df4614b55fb5fae142b98fbc40cc9034d23ce84e022554813446fbe" Oct 30 21:55:29.014171 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:29.014125 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"506ec9cf0df4614b55fb5fae142b98fbc40cc9034d23ce84e022554813446fbe"} err="failed to get container status \"506ec9cf0df4614b55fb5fae142b98fbc40cc9034d23ce84e022554813446fbe\": rpc error: code = NotFound desc = could not find container \"506ec9cf0df4614b55fb5fae142b98fbc40cc9034d23ce84e022554813446fbe\": container with ID starting with 506ec9cf0df4614b55fb5fae142b98fbc40cc9034d23ce84e022554813446fbe not found: ID does not exist" Oct 30 21:55:29.180515 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:29.180425 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2"] Oct 30 21:55:29.227049 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:29.227013 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-71aab5-openshift-default-6f49bc5f57-j7kf2"] Oct 30 21:55:30.577788 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:30.577759 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52c80c2c-2622-43a7-ab5c-bea1f6089c90" path="/var/lib/kubelet/pods/52c80c2c-2622-43a7-ab5c-bea1f6089c90/volumes" Oct 30 21:55:44.266502 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:44.266464 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:55:47.041121 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:47.041086 4560 scope.go:122] "RemoveContainer" containerID="3399edf97812039a0448a36c21245dc48cee4354065a5019738f5165b3e9ec41" Oct 30 21:55:47.050145 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:47.050115 4560 scope.go:122] "RemoveContainer" containerID="69d6f90cad9a8ed49db17092576f2025cdab07d8156f36ab2858c7857d050935" Oct 30 21:55:49.732157 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:55:49.732082 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:55:56.234333 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:56.234280 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj"] Oct 30 21:55:56.234871 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:56.234809 4560 cpu_manager.go:414] "containerMap: removing container" podUID="52c80c2c-2622-43a7-ab5c-bea1f6089c90" containerName="istio-proxy" Oct 30 21:55:56.234871 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:56.234822 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52c80c2c-2622-43a7-ab5c-bea1f6089c90" containerName="istio-proxy" Oct 30 21:55:56.234871 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:56.234847 4560 cpu_manager.go:414] "containerMap: removing container" podUID="9214c76a-32cc-45de-8130-a22060868346" containerName="revoke-keys" Oct 30 21:55:56.234871 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:56.234852 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9214c76a-32cc-45de-8130-a22060868346" containerName="revoke-keys" Oct 30 21:55:56.235000 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:56.234925 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="52c80c2c-2622-43a7-ab5c-bea1f6089c90" containerName="istio-proxy" Oct 30 21:55:56.235000 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:56.234936 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="9214c76a-32cc-45de-8130-a22060868346" containerName="revoke-keys" Oct 30 21:55:56.239774 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:56.239750 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" Oct 30 21:55:56.294384 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:56.289748 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c4e874-openshift-default-dockercfg-cq2qw\"" Oct 30 21:55:56.294578 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:56.294483 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-istio-podinfo\") pod \"e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj\" (UID: \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\") " pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" Oct 30 21:55:56.294578 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:56.294541 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-istio-envoy\") pod \"e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj\" (UID: \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\") " pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" Oct 30 21:55:56.294719 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:56.294611 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-workload-socket\") pod \"e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj\" (UID: \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\") " pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" Oct 30 21:55:56.294777 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:56.294743 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-credential-socket\") pod \"e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj\" (UID: \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\") " pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" Oct 30 21:55:56.294836 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:56.294771 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km9v9\" (UniqueName: \"kubernetes.io/projected/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-kube-api-access-km9v9\") pod \"e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj\" (UID: \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\") " pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" Oct 30 21:55:56.294892 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:56.294874 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-istio-data\") pod \"e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj\" (UID: \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\") " pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" Oct 30 21:55:56.294953 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:56.294938 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-istiod-ca-cert\") pod \"e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj\" (UID: \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\") " pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" Oct 30 21:55:56.295010 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:56.294971 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-workload-certs\") pod \"e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj\" (UID: \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\") " pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" Oct 30 21:55:56.295083 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:56.295062 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-istio-token\") pod \"e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj\" (UID: \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\") " pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" Oct 30 21:55:56.360223 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:56.360186 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj"] Oct 30 21:55:56.404804 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:56.403229 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-istio-envoy\") pod \"e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj\" (UID: \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\") " pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" Oct 30 21:55:56.405112 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:56.405085 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-workload-socket\") pod \"e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj\" (UID: \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\") " pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" Oct 30 21:55:56.405366 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:56.405351 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-credential-socket\") pod \"e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj\" (UID: \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\") " pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" Oct 30 21:55:56.405469 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:56.405457 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km9v9\" (UniqueName: \"kubernetes.io/projected/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-kube-api-access-km9v9\") pod \"e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj\" (UID: \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\") " pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" Oct 30 21:55:56.405617 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:56.405603 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-istio-data\") pod \"e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj\" (UID: \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\") " pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" Oct 30 21:55:56.405774 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:56.405763 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-istiod-ca-cert\") pod \"e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj\" (UID: \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\") " pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" Oct 30 21:55:56.405880 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:56.405869 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-workload-certs\") pod \"e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj\" (UID: \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\") " pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" Oct 30 21:55:56.405972 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:56.405962 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-istio-token\") pod \"e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj\" (UID: \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\") " pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" Oct 30 21:55:56.406095 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:56.406084 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-istio-podinfo\") pod \"e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj\" (UID: \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\") " pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" Oct 30 21:55:56.428412 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:56.413868 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-credential-socket\") pod \"e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj\" (UID: \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\") " pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" Oct 30 21:55:56.428412 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:56.414125 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-workload-socket\") pod \"e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj\" (UID: \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\") " pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" Oct 30 21:55:56.428412 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:56.414356 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-workload-certs\") pod \"e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj\" (UID: \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\") " pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" Oct 30 21:55:56.428412 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:56.414997 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-istiod-ca-cert\") pod \"e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj\" (UID: \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\") " pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" Oct 30 21:55:56.428725 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:56.428600 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-istio-podinfo\") pod \"e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj\" (UID: \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\") " pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" Oct 30 21:55:56.429251 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:56.429225 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-istio-data\") pod \"e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj\" (UID: \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\") " pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" Oct 30 21:55:56.430413 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:56.430363 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-istio-envoy\") pod \"e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj\" (UID: \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\") " pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" Oct 30 21:55:56.518807 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:56.518690 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-istio-token\") pod \"e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj\" (UID: \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\") " pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" Oct 30 21:55:56.518807 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:56.518796 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km9v9\" (UniqueName: \"kubernetes.io/projected/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-kube-api-access-km9v9\") pod \"e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj\" (UID: \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\") " pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" Oct 30 21:55:56.558363 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:56.558258 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" Oct 30 21:55:57.091038 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:57.091001 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj"] Oct 30 21:55:57.096229 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:55:57.096204 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64e8e251_1ae0_42cc_b46b_e8f11b3395ff.slice/crio-9418b2301eabdc205ed9076c0467f099fec9edac673815698d4e5f547e6264a5 WatchSource:0}: Error finding container 9418b2301eabdc205ed9076c0467f099fec9edac673815698d4e5f547e6264a5: Status 404 returned error can't find the container with id 9418b2301eabdc205ed9076c0467f099fec9edac673815698d4e5f547e6264a5 Oct 30 21:55:57.099387 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:57.099330 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:55:57.099475 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:57.099442 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:55:57.099514 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:57.099488 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:55:57.295321 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:57.295281 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" event={"ID":"64e8e251-1ae0-42cc-b46b-e8f11b3395ff","Type":"ContainerStarted","Data":"601da35e07d40b926692b97ec4f18d9ce1169c75d199608c6fb69fb27769b567"} Oct 30 21:55:57.295821 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:57.295442 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" event={"ID":"64e8e251-1ae0-42cc-b46b-e8f11b3395ff","Type":"ContainerStarted","Data":"9418b2301eabdc205ed9076c0467f099fec9edac673815698d4e5f547e6264a5"} Oct 30 21:55:57.416394 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:57.415902 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" podStartSLOduration=1.415883789 podStartE2EDuration="1.415883789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:57.415530332 +0000 UTC m=+2532.970230109" watchObservedRunningTime="2026-10-30 21:55:57.415883789 +0000 UTC m=+2532.970583567" Oct 30 21:55:57.563368 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:57.562476 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" Oct 30 21:55:58.565416 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:58.565382 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" Oct 30 21:55:59.308201 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:59.308166 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" Oct 30 21:55:59.309046 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:55:59.309021 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" Oct 30 21:56:14.510364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:14.509376 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76a9fb50-g4z94"] Oct 30 21:56:14.531198 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:14.522674 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a9fb50-g4z94" Oct 30 21:56:14.568582 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:14.567422 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76a9fb50-g4z94\" (UID: \"317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a9fb50-g4z94" Oct 30 21:56:14.568582 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:14.567558 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddz4b\" (UniqueName: \"kubernetes.io/projected/317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51-kube-api-access-ddz4b\") pod \"maas-api-revoke-sub-76a9fb50-g4z94\" (UID: \"317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a9fb50-g4z94" Oct 30 21:56:14.568582 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:14.567633 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76a9fb50-g4z94\" (UID: \"317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a9fb50-g4z94" Oct 30 21:56:14.575548 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:14.575246 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:56:14.577162 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:14.577134 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:56:14.578126 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:14.578086 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:56:14.581297 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:14.581270 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76a9fb50-g4z94"] Oct 30 21:56:14.623096 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:14.623059 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:56:14.669410 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:14.669376 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76a9fb50-g4z94\" (UID: \"317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a9fb50-g4z94" Oct 30 21:56:14.669624 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:14.669479 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddz4b\" (UniqueName: \"kubernetes.io/projected/317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51-kube-api-access-ddz4b\") pod \"maas-api-revoke-sub-76a9fb50-g4z94\" (UID: \"317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a9fb50-g4z94" Oct 30 21:56:14.669624 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:14.669560 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76a9fb50-g4z94\" (UID: \"317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a9fb50-g4z94" Oct 30 21:56:14.670301 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:14.670273 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76a9fb50-g4z94\" (UID: \"317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a9fb50-g4z94" Oct 30 21:56:14.671882 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:14.671865 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76a9fb50-g4z94\" (UID: \"317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a9fb50-g4z94" Oct 30 21:56:14.730684 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:14.730651 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddz4b\" (UniqueName: \"kubernetes.io/projected/317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51-kube-api-access-ddz4b\") pod \"maas-api-revoke-sub-76a9fb50-g4z94\" (UID: \"317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a9fb50-g4z94" Oct 30 21:56:14.845585 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:14.845505 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a9fb50-g4z94" Oct 30 21:56:15.229094 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:15.229056 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76a9fb50-g4z94"] Oct 30 21:56:15.234515 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:56:15.234484 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod317f75b4_a729_4a7b_8bc4_a5d0cbe4ac51.slice/crio-920e04ff6938e19a2a765dd50a64159ebd391d996a92d5e079d74f9c8518b28d WatchSource:0}: Error finding container 920e04ff6938e19a2a765dd50a64159ebd391d996a92d5e079d74f9c8518b28d: Status 404 returned error can't find the container with id 920e04ff6938e19a2a765dd50a64159ebd391d996a92d5e079d74f9c8518b28d Oct 30 21:56:15.455276 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:15.455245 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a9fb50-g4z94" event={"ID":"317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51","Type":"ContainerStarted","Data":"920e04ff6938e19a2a765dd50a64159ebd391d996a92d5e079d74f9c8518b28d"} Oct 30 21:56:16.463466 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:16.463381 4560 generic.go:356] "Generic (PLEG): container finished" podID="317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51" containerID="a6da00c9b2007936b529896fe61a57bfab83a3edabc837e94296ba99306a37df" exitCode=0 Oct 30 21:56:16.463830 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:16.463468 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a9fb50-g4z94" event={"ID":"317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51","Type":"ContainerDied","Data":"a6da00c9b2007936b529896fe61a57bfab83a3edabc837e94296ba99306a37df"} Oct 30 21:56:17.642042 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:17.642017 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a9fb50-g4z94" Oct 30 21:56:17.713617 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:17.713583 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51-maas-api-openshift-ca\") pod \"317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51\" (UID: \"317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51\") " Oct 30 21:56:17.713752 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:17.713669 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51-kube-api-access-ddz4b\" (UniqueName: \"kubernetes.io/projected/317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51-kube-api-access-ddz4b\") pod \"317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51\" (UID: \"317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51\") " Oct 30 21:56:17.713871 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:17.713858 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51-maas-api-xks-ca\") pod \"317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51\" (UID: \"317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51\") " Oct 30 21:56:17.713955 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:17.713930 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51-maas-api-openshift-ca" pod "317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51" (UID: "317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:17.714236 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:17.714221 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:17.715739 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:17.715716 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51-kube-api-access-ddz4b" pod "317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51" (UID: "317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51"). InnerVolumeSpecName "kube-api-access-ddz4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:17.715837 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:17.715768 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51-maas-api-xks-ca" pod "317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51" (UID: "317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:17.815264 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:17.815175 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:17.815264 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:17.815215 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddz4b\" (UniqueName: \"kubernetes.io/projected/317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51-kube-api-access-ddz4b\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:18.483962 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:18.483927 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a9fb50-g4z94" event={"ID":"317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51","Type":"ContainerDied","Data":"920e04ff6938e19a2a765dd50a64159ebd391d996a92d5e079d74f9c8518b28d"} Oct 30 21:56:18.484149 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:18.483975 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="920e04ff6938e19a2a765dd50a64159ebd391d996a92d5e079d74f9c8518b28d" Oct 30 21:56:18.484149 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:18.483938 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a9fb50-g4z94" Oct 30 21:56:19.686360 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:56:19.686304 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:56:26.689273 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:26.689240 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:56:27.051411 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:27.051373 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-655c1d85-gm7jv"] Oct 30 21:56:27.061919 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:27.061892 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-655c1d85-gm7jv"] Oct 30 21:56:28.476484 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:28.476447 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23808097-4s8n8"] Oct 30 21:56:28.476997 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:28.476989 4560 cpu_manager.go:414] "containerMap: removing container" podUID="317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51" containerName="revoke-keys" Oct 30 21:56:28.477063 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:28.477000 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51" containerName="revoke-keys" Oct 30 21:56:28.477102 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:28.477076 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51" containerName="revoke-keys" Oct 30 21:56:28.480320 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:28.480302 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23808097-4s8n8" Oct 30 21:56:28.507288 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:28.507258 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhbh5\" (UniqueName: \"kubernetes.io/projected/32d2830c-c92e-488a-a542-3eb9145cceab-kube-api-access-vhbh5\") pod \"maas-api-revoke-sub-23808097-4s8n8\" (UID: \"32d2830c-c92e-488a-a542-3eb9145cceab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23808097-4s8n8" Oct 30 21:56:28.507467 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:28.507448 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32d2830c-c92e-488a-a542-3eb9145cceab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23808097-4s8n8\" (UID: \"32d2830c-c92e-488a-a542-3eb9145cceab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23808097-4s8n8" Oct 30 21:56:28.507524 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:28.507513 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32d2830c-c92e-488a-a542-3eb9145cceab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23808097-4s8n8\" (UID: \"32d2830c-c92e-488a-a542-3eb9145cceab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23808097-4s8n8" Oct 30 21:56:28.530061 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:28.530033 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:56:28.530167 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:28.530132 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:56:28.576961 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:28.576906 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca7f9e19-a746-42c9-89fd-3361c8684774" path="/var/lib/kubelet/pods/ca7f9e19-a746-42c9-89fd-3361c8684774/volumes" Oct 30 21:56:28.577156 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:28.577117 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:56:28.577284 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:28.577268 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:56:28.580288 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:28.580263 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23808097-4s8n8"] Oct 30 21:56:28.608986 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:28.608961 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhbh5\" (UniqueName: \"kubernetes.io/projected/32d2830c-c92e-488a-a542-3eb9145cceab-kube-api-access-vhbh5\") pod \"maas-api-revoke-sub-23808097-4s8n8\" (UID: \"32d2830c-c92e-488a-a542-3eb9145cceab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23808097-4s8n8" Oct 30 21:56:28.609142 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:28.609091 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32d2830c-c92e-488a-a542-3eb9145cceab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23808097-4s8n8\" (UID: \"32d2830c-c92e-488a-a542-3eb9145cceab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23808097-4s8n8" Oct 30 21:56:28.609210 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:28.609187 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32d2830c-c92e-488a-a542-3eb9145cceab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23808097-4s8n8\" (UID: \"32d2830c-c92e-488a-a542-3eb9145cceab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23808097-4s8n8" Oct 30 21:56:28.609809 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:28.609785 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32d2830c-c92e-488a-a542-3eb9145cceab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23808097-4s8n8\" (UID: \"32d2830c-c92e-488a-a542-3eb9145cceab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23808097-4s8n8" Oct 30 21:56:28.611598 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:28.611575 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32d2830c-c92e-488a-a542-3eb9145cceab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23808097-4s8n8\" (UID: \"32d2830c-c92e-488a-a542-3eb9145cceab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23808097-4s8n8" Oct 30 21:56:28.666683 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:28.666653 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhbh5\" (UniqueName: \"kubernetes.io/projected/32d2830c-c92e-488a-a542-3eb9145cceab-kube-api-access-vhbh5\") pod \"maas-api-revoke-sub-23808097-4s8n8\" (UID: \"32d2830c-c92e-488a-a542-3eb9145cceab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23808097-4s8n8" Oct 30 21:56:28.793802 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:28.793725 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23808097-4s8n8" Oct 30 21:56:29.170504 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:29.170415 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23808097-4s8n8"] Oct 30 21:56:29.175422 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:56:29.175396 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32d2830c_c92e_488a_a542_3eb9145cceab.slice/crio-f1fa94adb45b369ee59160c2d3d173906c5dbdb73539fc8836473db7111946e7 WatchSource:0}: Error finding container f1fa94adb45b369ee59160c2d3d173906c5dbdb73539fc8836473db7111946e7: Status 404 returned error can't find the container with id f1fa94adb45b369ee59160c2d3d173906c5dbdb73539fc8836473db7111946e7 Oct 30 21:56:29.583945 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:29.583905 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23808097-4s8n8" event={"ID":"32d2830c-c92e-488a-a542-3eb9145cceab","Type":"ContainerStarted","Data":"6639cbbd016b9d7d69eb3d4a5bcb585552aeae0cb56eb8c5d0c1c57e0bacc78e"} Oct 30 21:56:29.584434 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:29.583964 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23808097-4s8n8" event={"ID":"32d2830c-c92e-488a-a542-3eb9145cceab","Type":"ContainerStarted","Data":"f1fa94adb45b369ee59160c2d3d173906c5dbdb73539fc8836473db7111946e7"} Oct 30 21:56:29.695907 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:29.695805 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23808097-4s8n8" podStartSLOduration=1.69579125 podStartE2EDuration="1.69579125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:29.693756419 +0000 UTC m=+2565.248456196" watchObservedRunningTime="2026-10-30 21:56:29.69579125 +0000 UTC m=+2565.250491026" Oct 30 21:56:30.590638 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:30.590610 4560 generic.go:356] "Generic (PLEG): container finished" podID="32d2830c-c92e-488a-a542-3eb9145cceab" containerID="6639cbbd016b9d7d69eb3d4a5bcb585552aeae0cb56eb8c5d0c1c57e0bacc78e" exitCode=0 Oct 30 21:56:30.591051 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:30.590698 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23808097-4s8n8" event={"ID":"32d2830c-c92e-488a-a542-3eb9145cceab","Type":"ContainerDied","Data":"6639cbbd016b9d7d69eb3d4a5bcb585552aeae0cb56eb8c5d0c1c57e0bacc78e"} Oct 30 21:56:31.749981 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:31.749953 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23808097-4s8n8" Oct 30 21:56:31.860962 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:31.860879 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32d2830c-c92e-488a-a542-3eb9145cceab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32d2830c-c92e-488a-a542-3eb9145cceab-maas-api-openshift-ca\") pod \"32d2830c-c92e-488a-a542-3eb9145cceab\" (UID: \"32d2830c-c92e-488a-a542-3eb9145cceab\") " Oct 30 21:56:31.861384 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:31.861323 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32d2830c-c92e-488a-a542-3eb9145cceab-kube-api-access-vhbh5\" (UniqueName: \"kubernetes.io/projected/32d2830c-c92e-488a-a542-3eb9145cceab-kube-api-access-vhbh5\") pod \"32d2830c-c92e-488a-a542-3eb9145cceab\" (UID: \"32d2830c-c92e-488a-a542-3eb9145cceab\") " Oct 30 21:56:31.861895 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:31.861863 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32d2830c-c92e-488a-a542-3eb9145cceab-maas-api-openshift-ca" pod "32d2830c-c92e-488a-a542-3eb9145cceab" (UID: "32d2830c-c92e-488a-a542-3eb9145cceab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:31.862298 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:31.861918 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32d2830c-c92e-488a-a542-3eb9145cceab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32d2830c-c92e-488a-a542-3eb9145cceab-maas-api-xks-ca\") pod \"32d2830c-c92e-488a-a542-3eb9145cceab\" (UID: \"32d2830c-c92e-488a-a542-3eb9145cceab\") " Oct 30 21:56:31.862554 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:31.862531 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32d2830c-c92e-488a-a542-3eb9145cceab-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:31.865549 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:31.865522 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32d2830c-c92e-488a-a542-3eb9145cceab-kube-api-access-vhbh5" pod "32d2830c-c92e-488a-a542-3eb9145cceab" (UID: "32d2830c-c92e-488a-a542-3eb9145cceab"). InnerVolumeSpecName "kube-api-access-vhbh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:31.865892 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:31.865866 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32d2830c-c92e-488a-a542-3eb9145cceab-maas-api-xks-ca" pod "32d2830c-c92e-488a-a542-3eb9145cceab" (UID: "32d2830c-c92e-488a-a542-3eb9145cceab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:31.963584 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:31.963552 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhbh5\" (UniqueName: \"kubernetes.io/projected/32d2830c-c92e-488a-a542-3eb9145cceab-kube-api-access-vhbh5\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:31.963584 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:31.963581 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32d2830c-c92e-488a-a542-3eb9145cceab-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:32.604159 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:32.604131 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23808097-4s8n8" Oct 30 21:56:32.604298 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:32.604153 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23808097-4s8n8" event={"ID":"32d2830c-c92e-488a-a542-3eb9145cceab","Type":"ContainerDied","Data":"f1fa94adb45b369ee59160c2d3d173906c5dbdb73539fc8836473db7111946e7"} Oct 30 21:56:32.604298 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:32.604200 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1fa94adb45b369ee59160c2d3d173906c5dbdb73539fc8836473db7111946e7" Oct 30 21:56:49.685915 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:56:49.685874 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:56:58.119417 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:58.119385 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7d7f57-c24mb"] Oct 30 21:56:58.119874 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:58.119861 4560 cpu_manager.go:414] "containerMap: removing container" podUID="32d2830c-c92e-488a-a542-3eb9145cceab" containerName="revoke-keys" Oct 30 21:56:58.119918 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:58.119874 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32d2830c-c92e-488a-a542-3eb9145cceab" containerName="revoke-keys" Oct 30 21:56:58.120009 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:58.119989 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="32d2830c-c92e-488a-a542-3eb9145cceab" containerName="revoke-keys" Oct 30 21:56:58.123557 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:58.123541 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7d7f57-c24mb" Oct 30 21:56:58.177361 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:58.175569 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:56:58.188614 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:58.179606 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:56:58.222788 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:58.222562 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:56:58.223162 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:58.223128 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww6nq\" (UniqueName: \"kubernetes.io/projected/ac521211-4a30-4704-832d-367e16899231-kube-api-access-ww6nq\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7d7f57-c24mb\" (UID: \"ac521211-4a30-4704-832d-367e16899231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7d7f57-c24mb" Oct 30 21:56:58.223273 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:58.223186 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac521211-4a30-4704-832d-367e16899231-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7d7f57-c24mb\" (UID: \"ac521211-4a30-4704-832d-367e16899231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7d7f57-c24mb" Oct 30 21:56:58.223450 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:58.223431 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac521211-4a30-4704-832d-367e16899231-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7d7f57-c24mb\" (UID: \"ac521211-4a30-4704-832d-367e16899231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7d7f57-c24mb" Oct 30 21:56:58.223645 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:58.223629 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:56:58.226306 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:58.226281 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7d7f57-c24mb"] Oct 30 21:56:58.325075 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:58.325041 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac521211-4a30-4704-832d-367e16899231-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7d7f57-c24mb\" (UID: \"ac521211-4a30-4704-832d-367e16899231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7d7f57-c24mb" Oct 30 21:56:58.325675 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:58.325649 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac521211-4a30-4704-832d-367e16899231-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7d7f57-c24mb\" (UID: \"ac521211-4a30-4704-832d-367e16899231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7d7f57-c24mb" Oct 30 21:56:58.325815 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:58.325777 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww6nq\" (UniqueName: \"kubernetes.io/projected/ac521211-4a30-4704-832d-367e16899231-kube-api-access-ww6nq\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7d7f57-c24mb\" (UID: \"ac521211-4a30-4704-832d-367e16899231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7d7f57-c24mb" Oct 30 21:56:58.325815 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:58.325804 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac521211-4a30-4704-832d-367e16899231-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7d7f57-c24mb\" (UID: \"ac521211-4a30-4704-832d-367e16899231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7d7f57-c24mb" Oct 30 21:56:58.402643 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:58.402568 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww6nq\" (UniqueName: \"kubernetes.io/projected/ac521211-4a30-4704-832d-367e16899231-kube-api-access-ww6nq\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7d7f57-c24mb\" (UID: \"ac521211-4a30-4704-832d-367e16899231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7d7f57-c24mb" Oct 30 21:56:58.403269 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:58.403244 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac521211-4a30-4704-832d-367e16899231-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7d7f57-c24mb\" (UID: \"ac521211-4a30-4704-832d-367e16899231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7d7f57-c24mb" Oct 30 21:56:58.489633 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:58.489589 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7d7f57-c24mb" Oct 30 21:56:59.282048 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:59.282014 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7d7f57-c24mb"] Oct 30 21:56:59.830614 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:56:59.830576 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7d7f57-c24mb" event={"ID":"ac521211-4a30-4704-832d-367e16899231","Type":"ContainerStarted","Data":"7e686a059e1fa484cbd9011c2020461a662c38365d1f58032ccdca27f4f13b40"} Oct 30 21:57:00.838408 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:00.838370 4560 generic.go:356] "Generic (PLEG): container finished" podID="ac521211-4a30-4704-832d-367e16899231" containerID="eb6eebf28355400d6615c9a7a1b8fb685e47af620c84f51016c2fa4a8b686048" exitCode=0 Oct 30 21:57:00.838862 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:00.838429 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7d7f57-c24mb" event={"ID":"ac521211-4a30-4704-832d-367e16899231","Type":"ContainerDied","Data":"eb6eebf28355400d6615c9a7a1b8fb685e47af620c84f51016c2fa4a8b686048"} Oct 30 21:57:01.999157 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:01.999132 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7d7f57-c24mb" Oct 30 21:57:02.054167 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:02.054134 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac521211-4a30-4704-832d-367e16899231-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac521211-4a30-4704-832d-367e16899231-maas-api-xks-ca\") pod \"ac521211-4a30-4704-832d-367e16899231\" (UID: \"ac521211-4a30-4704-832d-367e16899231\") " Oct 30 21:57:02.054332 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:02.054207 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac521211-4a30-4704-832d-367e16899231-kube-api-access-ww6nq\" (UniqueName: \"kubernetes.io/projected/ac521211-4a30-4704-832d-367e16899231-kube-api-access-ww6nq\") pod \"ac521211-4a30-4704-832d-367e16899231\" (UID: \"ac521211-4a30-4704-832d-367e16899231\") " Oct 30 21:57:02.054332 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:02.054289 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac521211-4a30-4704-832d-367e16899231-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac521211-4a30-4704-832d-367e16899231-maas-api-openshift-ca\") pod \"ac521211-4a30-4704-832d-367e16899231\" (UID: \"ac521211-4a30-4704-832d-367e16899231\") " Oct 30 21:57:02.054709 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:02.054682 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac521211-4a30-4704-832d-367e16899231-maas-api-openshift-ca" pod "ac521211-4a30-4704-832d-367e16899231" (UID: "ac521211-4a30-4704-832d-367e16899231"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:02.054950 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:02.054933 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac521211-4a30-4704-832d-367e16899231-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:02.056142 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:02.056121 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac521211-4a30-4704-832d-367e16899231-maas-api-xks-ca" pod "ac521211-4a30-4704-832d-367e16899231" (UID: "ac521211-4a30-4704-832d-367e16899231"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:02.056241 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:02.056228 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac521211-4a30-4704-832d-367e16899231-kube-api-access-ww6nq" pod "ac521211-4a30-4704-832d-367e16899231" (UID: "ac521211-4a30-4704-832d-367e16899231"). InnerVolumeSpecName "kube-api-access-ww6nq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:02.157029 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:02.156931 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac521211-4a30-4704-832d-367e16899231-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:02.157029 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:02.156974 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww6nq\" (UniqueName: \"kubernetes.io/projected/ac521211-4a30-4704-832d-367e16899231-kube-api-access-ww6nq\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:02.860351 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:02.860252 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7d7f57-c24mb" Oct 30 21:57:02.860351 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:02.860263 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7d7f57-c24mb" event={"ID":"ac521211-4a30-4704-832d-367e16899231","Type":"ContainerDied","Data":"7e686a059e1fa484cbd9011c2020461a662c38365d1f58032ccdca27f4f13b40"} Oct 30 21:57:02.860545 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:02.860364 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e686a059e1fa484cbd9011c2020461a662c38365d1f58032ccdca27f4f13b40" Oct 30 21:57:07.860275 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:07.860237 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv"] Oct 30 21:57:07.860748 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:07.860491 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" podUID="57d2540b-40a0-4326-a4af-57466f9abbae" containerName="istio-proxy" containerID="cri-o://b04c6444b1f6db659f1972b7786433757c02ecd83449a2cd3a2120d439607b8c" gracePeriod=30 Oct 30 21:57:07.909023 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:07.908983 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2"] Oct 30 21:57:07.909295 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:07.909242 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" podUID="a5e17746-9f03-4857-b551-bc5625cd5b67" containerName="istio-proxy" containerID="cri-o://c7f5ea5b89b2db32d5df33653b252331f4a02ba56ade1ce3de01a176d4807c83" gracePeriod=30 Oct 30 21:57:08.058688 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:08.058650 4560 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:57:08.058853 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:08.058702 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" podUID="a5e17746-9f03-4857-b551-bc5625cd5b67" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:57:09.431136 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:09.431101 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc"] Oct 30 21:57:09.431592 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:09.431305 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" podUID="1423e26a-6610-4adc-8a01-ebd58407a713" containerName="istio-proxy" containerID="cri-o://4524eaa22bbcb51637ed535f429ebf5cd042898e305a6184e06271ccaf7228a6" gracePeriod=30 Oct 30 21:57:10.735057 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:10.735022 4560 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:57:10.735470 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:10.735088 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" podUID="57d2540b-40a0-4326-a4af-57466f9abbae" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:57:11.246260 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:11.246227 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-737a64-htl7r"] Oct 30 21:57:11.246804 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:11.246781 4560 cpu_manager.go:414] "containerMap: removing container" podUID="ac521211-4a30-4704-832d-367e16899231" containerName="revoke-keys" Oct 30 21:57:11.246804 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:11.246799 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac521211-4a30-4704-832d-367e16899231" containerName="revoke-keys" Oct 30 21:57:11.246962 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:11.246923 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac521211-4a30-4704-832d-367e16899231" containerName="revoke-keys" Oct 30 21:57:11.250232 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:11.250216 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-737a64-htl7r" Oct 30 21:57:11.307390 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:11.307358 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-737a64-htl7r"] Oct 30 21:57:11.350050 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:11.350021 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:57:11.350194 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:11.350058 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:57:11.350194 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:11.350033 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:57:11.350474 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:11.350455 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:57:11.382779 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:11.382748 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49cb202f-c2ed-4107-89e6-dd5c5adef978-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-737a64-htl7r\" (UID: \"49cb202f-c2ed-4107-89e6-dd5c5adef978\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-737a64-htl7r" Oct 30 21:57:11.382929 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:11.382909 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc7v2\" (UniqueName: \"kubernetes.io/projected/49cb202f-c2ed-4107-89e6-dd5c5adef978-kube-api-access-gc7v2\") pod \"maas-api-revoke-keys-e2e-api-a-737a64-htl7r\" (UID: \"49cb202f-c2ed-4107-89e6-dd5c5adef978\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-737a64-htl7r" Oct 30 21:57:11.382973 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:11.382962 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49cb202f-c2ed-4107-89e6-dd5c5adef978-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-737a64-htl7r\" (UID: \"49cb202f-c2ed-4107-89e6-dd5c5adef978\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-737a64-htl7r" Oct 30 21:57:11.483811 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:11.483770 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49cb202f-c2ed-4107-89e6-dd5c5adef978-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-737a64-htl7r\" (UID: \"49cb202f-c2ed-4107-89e6-dd5c5adef978\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-737a64-htl7r" Oct 30 21:57:11.483985 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:11.483894 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49cb202f-c2ed-4107-89e6-dd5c5adef978-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-737a64-htl7r\" (UID: \"49cb202f-c2ed-4107-89e6-dd5c5adef978\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-737a64-htl7r" Oct 30 21:57:11.484032 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:11.484007 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc7v2\" (UniqueName: \"kubernetes.io/projected/49cb202f-c2ed-4107-89e6-dd5c5adef978-kube-api-access-gc7v2\") pod \"maas-api-revoke-keys-e2e-api-a-737a64-htl7r\" (UID: \"49cb202f-c2ed-4107-89e6-dd5c5adef978\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-737a64-htl7r" Oct 30 21:57:11.484587 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:11.484558 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49cb202f-c2ed-4107-89e6-dd5c5adef978-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-737a64-htl7r\" (UID: \"49cb202f-c2ed-4107-89e6-dd5c5adef978\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-737a64-htl7r" Oct 30 21:57:11.486044 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:11.486028 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49cb202f-c2ed-4107-89e6-dd5c5adef978-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-737a64-htl7r\" (UID: \"49cb202f-c2ed-4107-89e6-dd5c5adef978\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-737a64-htl7r" Oct 30 21:57:11.541761 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:11.541711 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc7v2\" (UniqueName: \"kubernetes.io/projected/49cb202f-c2ed-4107-89e6-dd5c5adef978-kube-api-access-gc7v2\") pod \"maas-api-revoke-keys-e2e-api-a-737a64-htl7r\" (UID: \"49cb202f-c2ed-4107-89e6-dd5c5adef978\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-737a64-htl7r" Oct 30 21:57:11.563744 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:11.563724 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-737a64-htl7r" Oct 30 21:57:11.955166 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:11.955080 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-737a64-htl7r"] Oct 30 21:57:11.960077 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:57:11.960048 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49cb202f_c2ed_4107_89e6_dd5c5adef978.slice/crio-7aa38331edd30faf850665faf80697a09f24b99295f3dedce4b4e48c260a73a0 WatchSource:0}: Error finding container 7aa38331edd30faf850665faf80697a09f24b99295f3dedce4b4e48c260a73a0: Status 404 returned error can't find the container with id 7aa38331edd30faf850665faf80697a09f24b99295f3dedce4b4e48c260a73a0 Oct 30 21:57:12.725823 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:12.725795 4560 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:57:12.725978 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:12.725853 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" podUID="1423e26a-6610-4adc-8a01-ebd58407a713" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:57:12.955335 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:12.955307 4560 generic.go:356] "Generic (PLEG): container finished" podID="49cb202f-c2ed-4107-89e6-dd5c5adef978" containerID="50babdb387fe22d83b206da377e53fe5e8d81ccf75ea603fdca5ab79e0322fb2" exitCode=0 Oct 30 21:57:12.955669 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:12.955368 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-737a64-htl7r" event={"ID":"49cb202f-c2ed-4107-89e6-dd5c5adef978","Type":"ContainerDied","Data":"50babdb387fe22d83b206da377e53fe5e8d81ccf75ea603fdca5ab79e0322fb2"} Oct 30 21:57:12.955669 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:12.955431 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-737a64-htl7r" event={"ID":"49cb202f-c2ed-4107-89e6-dd5c5adef978","Type":"ContainerStarted","Data":"7aa38331edd30faf850665faf80697a09f24b99295f3dedce4b4e48c260a73a0"} Oct 30 21:57:13.125696 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.125666 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" Oct 30 21:57:13.159729 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.159703 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" Oct 30 21:57:13.206511 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.206477 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57d2540b-40a0-4326-a4af-57466f9abbae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57d2540b-40a0-4326-a4af-57466f9abbae-istiod-ca-cert\") pod \"57d2540b-40a0-4326-a4af-57466f9abbae\" (UID: \"57d2540b-40a0-4326-a4af-57466f9abbae\") " Oct 30 21:57:13.206708 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.206542 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57d2540b-40a0-4326-a4af-57466f9abbae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57d2540b-40a0-4326-a4af-57466f9abbae-istio-data\") pod \"57d2540b-40a0-4326-a4af-57466f9abbae\" (UID: \"57d2540b-40a0-4326-a4af-57466f9abbae\") " Oct 30 21:57:13.206708 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.206576 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57d2540b-40a0-4326-a4af-57466f9abbae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57d2540b-40a0-4326-a4af-57466f9abbae-istio-podinfo\") pod \"57d2540b-40a0-4326-a4af-57466f9abbae\" (UID: \"57d2540b-40a0-4326-a4af-57466f9abbae\") " Oct 30 21:57:13.206708 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.206660 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57d2540b-40a0-4326-a4af-57466f9abbae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57d2540b-40a0-4326-a4af-57466f9abbae-istio-envoy\") pod \"57d2540b-40a0-4326-a4af-57466f9abbae\" (UID: \"57d2540b-40a0-4326-a4af-57466f9abbae\") " Oct 30 21:57:13.206872 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.206744 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57d2540b-40a0-4326-a4af-57466f9abbae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d2540b-40a0-4326-a4af-57466f9abbae-workload-socket\") pod \"57d2540b-40a0-4326-a4af-57466f9abbae\" (UID: \"57d2540b-40a0-4326-a4af-57466f9abbae\") " Oct 30 21:57:13.206872 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.206832 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57d2540b-40a0-4326-a4af-57466f9abbae-istio-data" pod "57d2540b-40a0-4326-a4af-57466f9abbae" (UID: "57d2540b-40a0-4326-a4af-57466f9abbae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:13.206872 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.206856 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57d2540b-40a0-4326-a4af-57466f9abbae-kube-api-access-qs8r7\" (UniqueName: \"kubernetes.io/projected/57d2540b-40a0-4326-a4af-57466f9abbae-kube-api-access-qs8r7\") pod \"57d2540b-40a0-4326-a4af-57466f9abbae\" (UID: \"57d2540b-40a0-4326-a4af-57466f9abbae\") " Oct 30 21:57:13.207087 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.206870 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57d2540b-40a0-4326-a4af-57466f9abbae-istiod-ca-cert" pod "57d2540b-40a0-4326-a4af-57466f9abbae" (UID: "57d2540b-40a0-4326-a4af-57466f9abbae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:13.207087 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.206898 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57d2540b-40a0-4326-a4af-57466f9abbae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d2540b-40a0-4326-a4af-57466f9abbae-credential-socket\") pod \"57d2540b-40a0-4326-a4af-57466f9abbae\" (UID: \"57d2540b-40a0-4326-a4af-57466f9abbae\") " Oct 30 21:57:13.207087 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.206952 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57d2540b-40a0-4326-a4af-57466f9abbae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57d2540b-40a0-4326-a4af-57466f9abbae-workload-certs\") pod \"57d2540b-40a0-4326-a4af-57466f9abbae\" (UID: \"57d2540b-40a0-4326-a4af-57466f9abbae\") " Oct 30 21:57:13.207087 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.206984 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57d2540b-40a0-4326-a4af-57466f9abbae-istio-token\" (UniqueName: \"kubernetes.io/projected/57d2540b-40a0-4326-a4af-57466f9abbae-istio-token\") pod \"57d2540b-40a0-4326-a4af-57466f9abbae\" (UID: \"57d2540b-40a0-4326-a4af-57466f9abbae\") " Oct 30 21:57:13.207087 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.207025 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57d2540b-40a0-4326-a4af-57466f9abbae-workload-socket" pod "57d2540b-40a0-4326-a4af-57466f9abbae" (UID: "57d2540b-40a0-4326-a4af-57466f9abbae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:13.207302 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.207165 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57d2540b-40a0-4326-a4af-57466f9abbae-credential-socket" pod "57d2540b-40a0-4326-a4af-57466f9abbae" (UID: "57d2540b-40a0-4326-a4af-57466f9abbae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:13.207302 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.207192 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57d2540b-40a0-4326-a4af-57466f9abbae-workload-certs" pod "57d2540b-40a0-4326-a4af-57466f9abbae" (UID: "57d2540b-40a0-4326-a4af-57466f9abbae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:13.207696 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.207670 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57d2540b-40a0-4326-a4af-57466f9abbae-istiod-ca-cert\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:13.207816 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.207700 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57d2540b-40a0-4326-a4af-57466f9abbae-istio-data\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:13.207816 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.207717 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d2540b-40a0-4326-a4af-57466f9abbae-workload-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:13.207816 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.207731 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57d2540b-40a0-4326-a4af-57466f9abbae-credential-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:13.207816 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.207749 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57d2540b-40a0-4326-a4af-57466f9abbae-workload-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:13.208970 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.208946 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57d2540b-40a0-4326-a4af-57466f9abbae-istio-envoy" pod "57d2540b-40a0-4326-a4af-57466f9abbae" (UID: "57d2540b-40a0-4326-a4af-57466f9abbae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:13.209051 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.208977 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57d2540b-40a0-4326-a4af-57466f9abbae-kube-api-access-qs8r7" pod "57d2540b-40a0-4326-a4af-57466f9abbae" (UID: "57d2540b-40a0-4326-a4af-57466f9abbae"). InnerVolumeSpecName "kube-api-access-qs8r7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:13.209051 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.208980 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57d2540b-40a0-4326-a4af-57466f9abbae-istio-podinfo" pod "57d2540b-40a0-4326-a4af-57466f9abbae" (UID: "57d2540b-40a0-4326-a4af-57466f9abbae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:57:13.209205 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.209191 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57d2540b-40a0-4326-a4af-57466f9abbae-istio-token" pod "57d2540b-40a0-4326-a4af-57466f9abbae" (UID: "57d2540b-40a0-4326-a4af-57466f9abbae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:13.308697 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.308603 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5e17746-9f03-4857-b551-bc5625cd5b67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5e17746-9f03-4857-b551-bc5625cd5b67-istio-envoy\") pod \"a5e17746-9f03-4857-b551-bc5625cd5b67\" (UID: \"a5e17746-9f03-4857-b551-bc5625cd5b67\") " Oct 30 21:57:13.308697 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.308681 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5e17746-9f03-4857-b551-bc5625cd5b67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e17746-9f03-4857-b551-bc5625cd5b67-workload-socket\") pod \"a5e17746-9f03-4857-b551-bc5625cd5b67\" (UID: \"a5e17746-9f03-4857-b551-bc5625cd5b67\") " Oct 30 21:57:13.308935 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.308730 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5e17746-9f03-4857-b551-bc5625cd5b67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5e17746-9f03-4857-b551-bc5625cd5b67-istio-podinfo\") pod \"a5e17746-9f03-4857-b551-bc5625cd5b67\" (UID: \"a5e17746-9f03-4857-b551-bc5625cd5b67\") " Oct 30 21:57:13.308935 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.308809 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5e17746-9f03-4857-b551-bc5625cd5b67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5e17746-9f03-4857-b551-bc5625cd5b67-istiod-ca-cert\") pod \"a5e17746-9f03-4857-b551-bc5625cd5b67\" (UID: \"a5e17746-9f03-4857-b551-bc5625cd5b67\") " Oct 30 21:57:13.308935 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.308849 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5e17746-9f03-4857-b551-bc5625cd5b67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5e17746-9f03-4857-b551-bc5625cd5b67-istio-data\") pod \"a5e17746-9f03-4857-b551-bc5625cd5b67\" (UID: \"a5e17746-9f03-4857-b551-bc5625cd5b67\") " Oct 30 21:57:13.308935 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.308885 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5e17746-9f03-4857-b551-bc5625cd5b67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5e17746-9f03-4857-b551-bc5625cd5b67-workload-certs\") pod \"a5e17746-9f03-4857-b551-bc5625cd5b67\" (UID: \"a5e17746-9f03-4857-b551-bc5625cd5b67\") " Oct 30 21:57:13.309162 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.308987 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5e17746-9f03-4857-b551-bc5625cd5b67-kube-api-access-5xhfp\" (UniqueName: \"kubernetes.io/projected/a5e17746-9f03-4857-b551-bc5625cd5b67-kube-api-access-5xhfp\") pod \"a5e17746-9f03-4857-b551-bc5625cd5b67\" (UID: \"a5e17746-9f03-4857-b551-bc5625cd5b67\") " Oct 30 21:57:13.309162 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.308993 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e17746-9f03-4857-b551-bc5625cd5b67-workload-socket" pod "a5e17746-9f03-4857-b551-bc5625cd5b67" (UID: "a5e17746-9f03-4857-b551-bc5625cd5b67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:13.309162 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.309061 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5e17746-9f03-4857-b551-bc5625cd5b67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e17746-9f03-4857-b551-bc5625cd5b67-credential-socket\") pod \"a5e17746-9f03-4857-b551-bc5625cd5b67\" (UID: \"a5e17746-9f03-4857-b551-bc5625cd5b67\") " Oct 30 21:57:13.309162 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.309131 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5e17746-9f03-4857-b551-bc5625cd5b67-istio-token\" (UniqueName: \"kubernetes.io/projected/a5e17746-9f03-4857-b551-bc5625cd5b67-istio-token\") pod \"a5e17746-9f03-4857-b551-bc5625cd5b67\" (UID: \"a5e17746-9f03-4857-b551-bc5625cd5b67\") " Oct 30 21:57:13.309470 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.309400 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e17746-9f03-4857-b551-bc5625cd5b67-credential-socket" pod "a5e17746-9f03-4857-b551-bc5625cd5b67" (UID: "a5e17746-9f03-4857-b551-bc5625cd5b67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:13.309470 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.309424 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5e17746-9f03-4857-b551-bc5625cd5b67-istiod-ca-cert" pod "a5e17746-9f03-4857-b551-bc5625cd5b67" (UID: "a5e17746-9f03-4857-b551-bc5625cd5b67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:13.309836 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.309811 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e17746-9f03-4857-b551-bc5625cd5b67-istio-data" pod "a5e17746-9f03-4857-b551-bc5625cd5b67" (UID: "a5e17746-9f03-4857-b551-bc5625cd5b67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:13.309983 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.309846 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e17746-9f03-4857-b551-bc5625cd5b67-workload-certs" pod "a5e17746-9f03-4857-b551-bc5625cd5b67" (UID: "a5e17746-9f03-4857-b551-bc5625cd5b67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:13.310084 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.309871 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs8r7\" (UniqueName: \"kubernetes.io/projected/57d2540b-40a0-4326-a4af-57466f9abbae-kube-api-access-qs8r7\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:13.310229 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.310098 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57d2540b-40a0-4326-a4af-57466f9abbae-istio-token\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:13.310229 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.310118 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e17746-9f03-4857-b551-bc5625cd5b67-workload-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:13.310229 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.310133 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5e17746-9f03-4857-b551-bc5625cd5b67-istiod-ca-cert\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:13.310229 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.310149 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57d2540b-40a0-4326-a4af-57466f9abbae-istio-podinfo\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:13.310229 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.310164 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57d2540b-40a0-4326-a4af-57466f9abbae-istio-envoy\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:13.310229 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.310179 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e17746-9f03-4857-b551-bc5625cd5b67-credential-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:13.311606 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.311572 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5e17746-9f03-4857-b551-bc5625cd5b67-kube-api-access-5xhfp" pod "a5e17746-9f03-4857-b551-bc5625cd5b67" (UID: "a5e17746-9f03-4857-b551-bc5625cd5b67"). InnerVolumeSpecName "kube-api-access-5xhfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:13.311718 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.311677 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e17746-9f03-4857-b551-bc5625cd5b67-istio-envoy" pod "a5e17746-9f03-4857-b551-bc5625cd5b67" (UID: "a5e17746-9f03-4857-b551-bc5625cd5b67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:13.311718 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.311693 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5e17746-9f03-4857-b551-bc5625cd5b67-istio-podinfo" pod "a5e17746-9f03-4857-b551-bc5625cd5b67" (UID: "a5e17746-9f03-4857-b551-bc5625cd5b67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:57:13.312479 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.312457 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5e17746-9f03-4857-b551-bc5625cd5b67-istio-token" pod "a5e17746-9f03-4857-b551-bc5625cd5b67" (UID: "a5e17746-9f03-4857-b551-bc5625cd5b67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:13.411403 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.411368 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5e17746-9f03-4857-b551-bc5625cd5b67-istio-podinfo\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:13.411403 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.411401 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5e17746-9f03-4857-b551-bc5625cd5b67-istio-data\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:13.411614 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.411413 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5e17746-9f03-4857-b551-bc5625cd5b67-workload-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:13.411614 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.411425 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xhfp\" (UniqueName: \"kubernetes.io/projected/a5e17746-9f03-4857-b551-bc5625cd5b67-kube-api-access-5xhfp\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:13.411614 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.411439 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5e17746-9f03-4857-b551-bc5625cd5b67-istio-token\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:13.411614 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:13.411450 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5e17746-9f03-4857-b551-bc5625cd5b67-istio-envoy\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:14.052566 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.052520 4560 generic.go:356] "Generic (PLEG): container finished" podID="a5e17746-9f03-4857-b551-bc5625cd5b67" containerID="c7f5ea5b89b2db32d5df33653b252331f4a02ba56ade1ce3de01a176d4807c83" exitCode=0 Oct 30 21:57:14.053241 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.053212 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" event={"ID":"a5e17746-9f03-4857-b551-bc5625cd5b67","Type":"ContainerDied","Data":"c7f5ea5b89b2db32d5df33653b252331f4a02ba56ade1ce3de01a176d4807c83"} Oct 30 21:57:14.053476 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.053458 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" event={"ID":"a5e17746-9f03-4857-b551-bc5625cd5b67","Type":"ContainerDied","Data":"8c6600b768b88a9c4c5a85c517cdb1c32727d5661f144645685ca69ad6ad8028"} Oct 30 21:57:14.053618 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.053605 4560 scope.go:122] "RemoveContainer" containerID="c7f5ea5b89b2db32d5df33653b252331f4a02ba56ade1ce3de01a176d4807c83" Oct 30 21:57:14.053885 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.053870 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2" Oct 30 21:57:14.074757 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.074721 4560 generic.go:356] "Generic (PLEG): container finished" podID="57d2540b-40a0-4326-a4af-57466f9abbae" containerID="b04c6444b1f6db659f1972b7786433757c02ecd83449a2cd3a2120d439607b8c" exitCode=0 Oct 30 21:57:14.075276 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.075038 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" Oct 30 21:57:14.077003 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.076955 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" event={"ID":"57d2540b-40a0-4326-a4af-57466f9abbae","Type":"ContainerDied","Data":"b04c6444b1f6db659f1972b7786433757c02ecd83449a2cd3a2120d439607b8c"} Oct 30 21:57:14.077084 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.077013 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv" event={"ID":"57d2540b-40a0-4326-a4af-57466f9abbae","Type":"ContainerDied","Data":"f2569acb84c371757485d7c7adb6ea2f9cd67daf696aca1c948dcb2c5ffb0edf"} Oct 30 21:57:14.090616 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.089133 4560 scope.go:122] "RemoveContainer" containerID="c7f5ea5b89b2db32d5df33653b252331f4a02ba56ade1ce3de01a176d4807c83" Oct 30 21:57:14.090616 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:57:14.089487 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7f5ea5b89b2db32d5df33653b252331f4a02ba56ade1ce3de01a176d4807c83\": container with ID starting with c7f5ea5b89b2db32d5df33653b252331f4a02ba56ade1ce3de01a176d4807c83 not found: ID does not exist" containerID="c7f5ea5b89b2db32d5df33653b252331f4a02ba56ade1ce3de01a176d4807c83" Oct 30 21:57:14.090616 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.089522 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7f5ea5b89b2db32d5df33653b252331f4a02ba56ade1ce3de01a176d4807c83"} err="failed to get container status \"c7f5ea5b89b2db32d5df33653b252331f4a02ba56ade1ce3de01a176d4807c83\": rpc error: code = NotFound desc = could not find container \"c7f5ea5b89b2db32d5df33653b252331f4a02ba56ade1ce3de01a176d4807c83\": container with ID starting with c7f5ea5b89b2db32d5df33653b252331f4a02ba56ade1ce3de01a176d4807c83 not found: ID does not exist" Oct 30 21:57:14.090616 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.089546 4560 scope.go:122] "RemoveContainer" containerID="b04c6444b1f6db659f1972b7786433757c02ecd83449a2cd3a2120d439607b8c" Oct 30 21:57:14.103624 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.103603 4560 scope.go:122] "RemoveContainer" containerID="b04c6444b1f6db659f1972b7786433757c02ecd83449a2cd3a2120d439607b8c" Oct 30 21:57:14.103934 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:57:14.103915 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b04c6444b1f6db659f1972b7786433757c02ecd83449a2cd3a2120d439607b8c\": container with ID starting with b04c6444b1f6db659f1972b7786433757c02ecd83449a2cd3a2120d439607b8c not found: ID does not exist" containerID="b04c6444b1f6db659f1972b7786433757c02ecd83449a2cd3a2120d439607b8c" Oct 30 21:57:14.104013 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.103944 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b04c6444b1f6db659f1972b7786433757c02ecd83449a2cd3a2120d439607b8c"} err="failed to get container status \"b04c6444b1f6db659f1972b7786433757c02ecd83449a2cd3a2120d439607b8c\": rpc error: code = NotFound desc = could not find container \"b04c6444b1f6db659f1972b7786433757c02ecd83449a2cd3a2120d439607b8c\": container with ID starting with b04c6444b1f6db659f1972b7786433757c02ecd83449a2cd3a2120d439607b8c not found: ID does not exist" Oct 30 21:57:14.232483 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.232449 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2"] Oct 30 21:57:14.243826 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.243794 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-s4qt2"] Oct 30 21:57:14.302404 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.302384 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-737a64-htl7r" Oct 30 21:57:14.353479 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.353422 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv"] Oct 30 21:57:14.368741 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.368718 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-mgqzv"] Oct 30 21:57:14.424886 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.424862 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49cb202f-c2ed-4107-89e6-dd5c5adef978-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49cb202f-c2ed-4107-89e6-dd5c5adef978-maas-api-xks-ca\") pod \"49cb202f-c2ed-4107-89e6-dd5c5adef978\" (UID: \"49cb202f-c2ed-4107-89e6-dd5c5adef978\") " Oct 30 21:57:14.424996 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.424954 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49cb202f-c2ed-4107-89e6-dd5c5adef978-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49cb202f-c2ed-4107-89e6-dd5c5adef978-maas-api-openshift-ca\") pod \"49cb202f-c2ed-4107-89e6-dd5c5adef978\" (UID: \"49cb202f-c2ed-4107-89e6-dd5c5adef978\") " Oct 30 21:57:14.425050 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.425033 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49cb202f-c2ed-4107-89e6-dd5c5adef978-kube-api-access-gc7v2\" (UniqueName: \"kubernetes.io/projected/49cb202f-c2ed-4107-89e6-dd5c5adef978-kube-api-access-gc7v2\") pod \"49cb202f-c2ed-4107-89e6-dd5c5adef978\" (UID: \"49cb202f-c2ed-4107-89e6-dd5c5adef978\") " Oct 30 21:57:14.425303 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.425282 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49cb202f-c2ed-4107-89e6-dd5c5adef978-maas-api-openshift-ca" pod "49cb202f-c2ed-4107-89e6-dd5c5adef978" (UID: "49cb202f-c2ed-4107-89e6-dd5c5adef978"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:14.425515 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.425499 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49cb202f-c2ed-4107-89e6-dd5c5adef978-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:14.426971 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.426956 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49cb202f-c2ed-4107-89e6-dd5c5adef978-maas-api-xks-ca" pod "49cb202f-c2ed-4107-89e6-dd5c5adef978" (UID: "49cb202f-c2ed-4107-89e6-dd5c5adef978"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:14.427093 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.427081 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49cb202f-c2ed-4107-89e6-dd5c5adef978-kube-api-access-gc7v2" pod "49cb202f-c2ed-4107-89e6-dd5c5adef978" (UID: "49cb202f-c2ed-4107-89e6-dd5c5adef978"). InnerVolumeSpecName "kube-api-access-gc7v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:14.526907 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.526880 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc7v2\" (UniqueName: \"kubernetes.io/projected/49cb202f-c2ed-4107-89e6-dd5c5adef978-kube-api-access-gc7v2\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:14.526907 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.526907 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49cb202f-c2ed-4107-89e6-dd5c5adef978-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:14.576640 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.576608 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57d2540b-40a0-4326-a4af-57466f9abbae" path="/var/lib/kubelet/pods/57d2540b-40a0-4326-a4af-57466f9abbae/volumes" Oct 30 21:57:14.577158 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.577136 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5e17746-9f03-4857-b551-bc5625cd5b67" path="/var/lib/kubelet/pods/a5e17746-9f03-4857-b551-bc5625cd5b67/volumes" Oct 30 21:57:14.702453 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.702429 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" Oct 30 21:57:14.834853 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.834816 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1423e26a-6610-4adc-8a01-ebd58407a713-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1423e26a-6610-4adc-8a01-ebd58407a713-istio-data\") pod \"1423e26a-6610-4adc-8a01-ebd58407a713\" (UID: \"1423e26a-6610-4adc-8a01-ebd58407a713\") " Oct 30 21:57:14.835009 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.834877 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1423e26a-6610-4adc-8a01-ebd58407a713-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1423e26a-6610-4adc-8a01-ebd58407a713-credential-socket\") pod \"1423e26a-6610-4adc-8a01-ebd58407a713\" (UID: \"1423e26a-6610-4adc-8a01-ebd58407a713\") " Oct 30 21:57:14.835009 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.834912 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1423e26a-6610-4adc-8a01-ebd58407a713-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1423e26a-6610-4adc-8a01-ebd58407a713-istio-podinfo\") pod \"1423e26a-6610-4adc-8a01-ebd58407a713\" (UID: \"1423e26a-6610-4adc-8a01-ebd58407a713\") " Oct 30 21:57:14.835009 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.834957 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1423e26a-6610-4adc-8a01-ebd58407a713-istio-token\" (UniqueName: \"kubernetes.io/projected/1423e26a-6610-4adc-8a01-ebd58407a713-istio-token\") pod \"1423e26a-6610-4adc-8a01-ebd58407a713\" (UID: \"1423e26a-6610-4adc-8a01-ebd58407a713\") " Oct 30 21:57:14.835196 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.835024 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1423e26a-6610-4adc-8a01-ebd58407a713-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1423e26a-6610-4adc-8a01-ebd58407a713-istio-envoy\") pod \"1423e26a-6610-4adc-8a01-ebd58407a713\" (UID: \"1423e26a-6610-4adc-8a01-ebd58407a713\") " Oct 30 21:57:14.835196 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.835127 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1423e26a-6610-4adc-8a01-ebd58407a713-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1423e26a-6610-4adc-8a01-ebd58407a713-workload-certs\") pod \"1423e26a-6610-4adc-8a01-ebd58407a713\" (UID: \"1423e26a-6610-4adc-8a01-ebd58407a713\") " Oct 30 21:57:14.835196 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.835125 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1423e26a-6610-4adc-8a01-ebd58407a713-istio-data" pod "1423e26a-6610-4adc-8a01-ebd58407a713" (UID: "1423e26a-6610-4adc-8a01-ebd58407a713"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:14.835196 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.835173 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1423e26a-6610-4adc-8a01-ebd58407a713-credential-socket" pod "1423e26a-6610-4adc-8a01-ebd58407a713" (UID: "1423e26a-6610-4adc-8a01-ebd58407a713"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:14.835468 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.835241 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1423e26a-6610-4adc-8a01-ebd58407a713-kube-api-access-gnpgg\" (UniqueName: \"kubernetes.io/projected/1423e26a-6610-4adc-8a01-ebd58407a713-kube-api-access-gnpgg\") pod \"1423e26a-6610-4adc-8a01-ebd58407a713\" (UID: \"1423e26a-6610-4adc-8a01-ebd58407a713\") " Oct 30 21:57:14.835468 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.835287 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1423e26a-6610-4adc-8a01-ebd58407a713-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1423e26a-6610-4adc-8a01-ebd58407a713-workload-socket\") pod \"1423e26a-6610-4adc-8a01-ebd58407a713\" (UID: \"1423e26a-6610-4adc-8a01-ebd58407a713\") " Oct 30 21:57:14.835468 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.835441 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1423e26a-6610-4adc-8a01-ebd58407a713-workload-certs" pod "1423e26a-6610-4adc-8a01-ebd58407a713" (UID: "1423e26a-6610-4adc-8a01-ebd58407a713"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:14.835640 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.835519 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1423e26a-6610-4adc-8a01-ebd58407a713-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1423e26a-6610-4adc-8a01-ebd58407a713-istiod-ca-cert\") pod \"1423e26a-6610-4adc-8a01-ebd58407a713\" (UID: \"1423e26a-6610-4adc-8a01-ebd58407a713\") " Oct 30 21:57:14.836128 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.836091 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1423e26a-6610-4adc-8a01-ebd58407a713-istio-data\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:14.836297 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.836131 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1423e26a-6610-4adc-8a01-ebd58407a713-credential-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:14.836297 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.836146 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1423e26a-6610-4adc-8a01-ebd58407a713-workload-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:14.836575 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.836555 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1423e26a-6610-4adc-8a01-ebd58407a713-istiod-ca-cert" pod "1423e26a-6610-4adc-8a01-ebd58407a713" (UID: "1423e26a-6610-4adc-8a01-ebd58407a713"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:14.836772 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.836750 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1423e26a-6610-4adc-8a01-ebd58407a713-workload-socket" pod "1423e26a-6610-4adc-8a01-ebd58407a713" (UID: "1423e26a-6610-4adc-8a01-ebd58407a713"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:14.845571 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.845468 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1423e26a-6610-4adc-8a01-ebd58407a713-istio-envoy" pod "1423e26a-6610-4adc-8a01-ebd58407a713" (UID: "1423e26a-6610-4adc-8a01-ebd58407a713"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:14.845571 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.845492 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1423e26a-6610-4adc-8a01-ebd58407a713-istio-token" pod "1423e26a-6610-4adc-8a01-ebd58407a713" (UID: "1423e26a-6610-4adc-8a01-ebd58407a713"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:14.845571 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.845533 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1423e26a-6610-4adc-8a01-ebd58407a713-kube-api-access-gnpgg" pod "1423e26a-6610-4adc-8a01-ebd58407a713" (UID: "1423e26a-6610-4adc-8a01-ebd58407a713"). InnerVolumeSpecName "kube-api-access-gnpgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:14.846725 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.846699 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1423e26a-6610-4adc-8a01-ebd58407a713-istio-podinfo" pod "1423e26a-6610-4adc-8a01-ebd58407a713" (UID: "1423e26a-6610-4adc-8a01-ebd58407a713"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:57:14.938604 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.938488 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1423e26a-6610-4adc-8a01-ebd58407a713-istiod-ca-cert\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:14.938604 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.938524 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1423e26a-6610-4adc-8a01-ebd58407a713-istio-podinfo\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:14.938604 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.938537 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1423e26a-6610-4adc-8a01-ebd58407a713-istio-token\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:14.938604 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.938550 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1423e26a-6610-4adc-8a01-ebd58407a713-istio-envoy\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:14.938604 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.938564 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnpgg\" (UniqueName: \"kubernetes.io/projected/1423e26a-6610-4adc-8a01-ebd58407a713-kube-api-access-gnpgg\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:14.938604 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:14.938578 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1423e26a-6610-4adc-8a01-ebd58407a713-workload-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:15.060869 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:15.060837 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29f04460-2xvqd"] Oct 30 21:57:15.068643 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:15.068619 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29f04460-2xvqd"] Oct 30 21:57:15.083155 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:15.083137 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-737a64-htl7r" Oct 30 21:57:15.083313 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:15.083172 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-737a64-htl7r" event={"ID":"49cb202f-c2ed-4107-89e6-dd5c5adef978","Type":"ContainerDied","Data":"7aa38331edd30faf850665faf80697a09f24b99295f3dedce4b4e48c260a73a0"} Oct 30 21:57:15.083313 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:15.083229 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7aa38331edd30faf850665faf80697a09f24b99295f3dedce4b4e48c260a73a0" Oct 30 21:57:15.084764 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:15.084738 4560 generic.go:356] "Generic (PLEG): container finished" podID="1423e26a-6610-4adc-8a01-ebd58407a713" containerID="4524eaa22bbcb51637ed535f429ebf5cd042898e305a6184e06271ccaf7228a6" exitCode=0 Oct 30 21:57:15.084885 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:15.084781 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" Oct 30 21:57:15.084885 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:15.084826 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" event={"ID":"1423e26a-6610-4adc-8a01-ebd58407a713","Type":"ContainerDied","Data":"4524eaa22bbcb51637ed535f429ebf5cd042898e305a6184e06271ccaf7228a6"} Oct 30 21:57:15.084885 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:15.084868 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc" event={"ID":"1423e26a-6610-4adc-8a01-ebd58407a713","Type":"ContainerDied","Data":"646161b1a1ee147dd7d7d8a5389dc034c717a1e81d19b2876ea4eb14e747cf9f"} Oct 30 21:57:15.085070 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:15.084890 4560 scope.go:122] "RemoveContainer" containerID="4524eaa22bbcb51637ed535f429ebf5cd042898e305a6184e06271ccaf7228a6" Oct 30 21:57:15.097397 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:15.097379 4560 scope.go:122] "RemoveContainer" containerID="4524eaa22bbcb51637ed535f429ebf5cd042898e305a6184e06271ccaf7228a6" Oct 30 21:57:15.097697 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:57:15.097677 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4524eaa22bbcb51637ed535f429ebf5cd042898e305a6184e06271ccaf7228a6\": container with ID starting with 4524eaa22bbcb51637ed535f429ebf5cd042898e305a6184e06271ccaf7228a6 not found: ID does not exist" containerID="4524eaa22bbcb51637ed535f429ebf5cd042898e305a6184e06271ccaf7228a6" Oct 30 21:57:15.097783 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:15.097704 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4524eaa22bbcb51637ed535f429ebf5cd042898e305a6184e06271ccaf7228a6"} err="failed to get container status \"4524eaa22bbcb51637ed535f429ebf5cd042898e305a6184e06271ccaf7228a6\": rpc error: code = NotFound desc = could not find container \"4524eaa22bbcb51637ed535f429ebf5cd042898e305a6184e06271ccaf7228a6\": container with ID starting with 4524eaa22bbcb51637ed535f429ebf5cd042898e305a6184e06271ccaf7228a6 not found: ID does not exist" Oct 30 21:57:15.322804 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:15.322769 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc"] Oct 30 21:57:15.332316 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:15.331223 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-2nxlc"] Oct 30 21:57:16.575624 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:16.575596 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1423e26a-6610-4adc-8a01-ebd58407a713" path="/var/lib/kubelet/pods/1423e26a-6610-4adc-8a01-ebd58407a713/volumes" Oct 30 21:57:16.576094 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:16.576032 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be68ac41-bb28-46e6-acf3-b880a29513d3" path="/var/lib/kubelet/pods/be68ac41-bb28-46e6-acf3-b880a29513d3/volumes" Oct 30 21:57:19.687188 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:57:19.687144 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:57:21.969246 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:21.969210 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2aa268-wtnzv"] Oct 30 21:57:21.969887 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:21.969870 4560 cpu_manager.go:414] "containerMap: removing container" podUID="49cb202f-c2ed-4107-89e6-dd5c5adef978" containerName="revoke-keys" Oct 30 21:57:21.969964 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:21.969889 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49cb202f-c2ed-4107-89e6-dd5c5adef978" containerName="revoke-keys" Oct 30 21:57:21.969964 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:21.969917 4560 cpu_manager.go:414] "containerMap: removing container" podUID="1423e26a-6610-4adc-8a01-ebd58407a713" containerName="istio-proxy" Oct 30 21:57:21.969964 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:21.969924 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1423e26a-6610-4adc-8a01-ebd58407a713" containerName="istio-proxy" Oct 30 21:57:21.969964 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:21.969949 4560 cpu_manager.go:414] "containerMap: removing container" podUID="a5e17746-9f03-4857-b551-bc5625cd5b67" containerName="istio-proxy" Oct 30 21:57:21.969964 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:21.969957 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5e17746-9f03-4857-b551-bc5625cd5b67" containerName="istio-proxy" Oct 30 21:57:21.970118 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:21.969985 4560 cpu_manager.go:414] "containerMap: removing container" podUID="57d2540b-40a0-4326-a4af-57466f9abbae" containerName="istio-proxy" Oct 30 21:57:21.970118 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:21.969993 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57d2540b-40a0-4326-a4af-57466f9abbae" containerName="istio-proxy" Oct 30 21:57:21.970118 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:21.970102 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="1423e26a-6610-4adc-8a01-ebd58407a713" containerName="istio-proxy" Oct 30 21:57:21.970118 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:21.970116 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5e17746-9f03-4857-b551-bc5625cd5b67" containerName="istio-proxy" Oct 30 21:57:21.970241 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:21.970130 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="57d2540b-40a0-4326-a4af-57466f9abbae" containerName="istio-proxy" Oct 30 21:57:21.970241 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:21.970142 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="49cb202f-c2ed-4107-89e6-dd5c5adef978" containerName="revoke-keys" Oct 30 21:57:21.974953 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:21.974931 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2aa268-wtnzv" Oct 30 21:57:22.028721 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:22.028687 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2aa268-wtnzv"] Oct 30 21:57:22.071108 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:22.071079 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:57:22.071108 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:22.071086 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:57:22.071318 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:22.071161 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:57:22.071318 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:22.071174 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:57:22.140665 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:22.140636 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgl77\" (UniqueName: \"kubernetes.io/projected/6fbeac30-9ae4-4163-8b26-edc5ec2099e2-kube-api-access-fgl77\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2aa268-wtnzv\" (UID: \"6fbeac30-9ae4-4163-8b26-edc5ec2099e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2aa268-wtnzv" Oct 30 21:57:22.140853 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:22.140700 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fbeac30-9ae4-4163-8b26-edc5ec2099e2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2aa268-wtnzv\" (UID: \"6fbeac30-9ae4-4163-8b26-edc5ec2099e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2aa268-wtnzv" Oct 30 21:57:22.140909 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:22.140899 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fbeac30-9ae4-4163-8b26-edc5ec2099e2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2aa268-wtnzv\" (UID: \"6fbeac30-9ae4-4163-8b26-edc5ec2099e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2aa268-wtnzv" Oct 30 21:57:22.241592 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:22.241569 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fbeac30-9ae4-4163-8b26-edc5ec2099e2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2aa268-wtnzv\" (UID: \"6fbeac30-9ae4-4163-8b26-edc5ec2099e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2aa268-wtnzv" Oct 30 21:57:22.241734 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:22.241605 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgl77\" (UniqueName: \"kubernetes.io/projected/6fbeac30-9ae4-4163-8b26-edc5ec2099e2-kube-api-access-fgl77\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2aa268-wtnzv\" (UID: \"6fbeac30-9ae4-4163-8b26-edc5ec2099e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2aa268-wtnzv" Oct 30 21:57:22.241734 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:22.241645 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fbeac30-9ae4-4163-8b26-edc5ec2099e2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2aa268-wtnzv\" (UID: \"6fbeac30-9ae4-4163-8b26-edc5ec2099e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2aa268-wtnzv" Oct 30 21:57:22.242303 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:22.242281 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fbeac30-9ae4-4163-8b26-edc5ec2099e2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2aa268-wtnzv\" (UID: \"6fbeac30-9ae4-4163-8b26-edc5ec2099e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2aa268-wtnzv" Oct 30 21:57:22.243906 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:22.243884 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fbeac30-9ae4-4163-8b26-edc5ec2099e2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2aa268-wtnzv\" (UID: \"6fbeac30-9ae4-4163-8b26-edc5ec2099e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2aa268-wtnzv" Oct 30 21:57:22.299825 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:22.299791 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgl77\" (UniqueName: \"kubernetes.io/projected/6fbeac30-9ae4-4163-8b26-edc5ec2099e2-kube-api-access-fgl77\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2aa268-wtnzv\" (UID: \"6fbeac30-9ae4-4163-8b26-edc5ec2099e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2aa268-wtnzv" Oct 30 21:57:22.588626 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:22.588560 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2aa268-wtnzv" Oct 30 21:57:22.963575 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:22.963505 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2aa268-wtnzv"] Oct 30 21:57:23.137839 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:23.137804 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2aa268-wtnzv" event={"ID":"6fbeac30-9ae4-4163-8b26-edc5ec2099e2","Type":"ContainerStarted","Data":"958bfd399c863caba8a151119f796865f21e44f065cf476b6e3615067930a1b1"} Oct 30 21:57:23.403366 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:23.403312 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6"] Oct 30 21:57:23.403561 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:23.403523 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" podUID="a0802666-5926-4a4f-a438-8e932494758c" containerName="istio-proxy" containerID="cri-o://9e12acd38ecb1e266db2ec5f3db45fce431c0094c6e7044baa990d1f514cec6f" gracePeriod=30 Oct 30 21:57:23.450635 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:23.450609 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb"] Oct 30 21:57:23.450842 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:23.450805 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" podUID="aaf8da80-7e8e-40a3-8c88-33b2325ef2aa" containerName="istio-proxy" containerID="cri-o://dc97903049a2d73dc0d3d2931402c73118f0019cf5f6396076ebeedd6f969433" gracePeriod=30 Oct 30 21:57:24.144908 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:24.144810 4560 generic.go:356] "Generic (PLEG): container finished" podID="6fbeac30-9ae4-4163-8b26-edc5ec2099e2" containerID="d0b9739a46201a3c194276e83be7def8ac065a7c348f7f603e9c4b8e794b1b45" exitCode=0 Oct 30 21:57:24.145332 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:24.144898 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2aa268-wtnzv" event={"ID":"6fbeac30-9ae4-4163-8b26-edc5ec2099e2","Type":"ContainerDied","Data":"d0b9739a46201a3c194276e83be7def8ac065a7c348f7f603e9c4b8e794b1b45"} Oct 30 21:57:24.613757 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:24.613726 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j"] Oct 30 21:57:24.614229 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:24.614199 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" podUID="408e0e40-4d43-4ef8-adb0-07f371a17dd8" containerName="istio-proxy" containerID="cri-o://e5fe40808d042be41381cb486e4fea21666d6ebdd287e51b10a4db08c289efb7" gracePeriod=30 Oct 30 21:57:25.066321 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:25.066289 4560 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:57:25.066501 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:25.066354 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" podUID="a0802666-5926-4a4f-a438-8e932494758c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:57:25.314500 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:25.314475 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2aa268-wtnzv" Oct 30 21:57:25.319810 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:25.319763 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fbeac30-9ae4-4163-8b26-edc5ec2099e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fbeac30-9ae4-4163-8b26-edc5ec2099e2-maas-api-xks-ca\") pod \"6fbeac30-9ae4-4163-8b26-edc5ec2099e2\" (UID: \"6fbeac30-9ae4-4163-8b26-edc5ec2099e2\") " Oct 30 21:57:25.319894 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:25.319833 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fbeac30-9ae4-4163-8b26-edc5ec2099e2-kube-api-access-fgl77\" (UniqueName: \"kubernetes.io/projected/6fbeac30-9ae4-4163-8b26-edc5ec2099e2-kube-api-access-fgl77\") pod \"6fbeac30-9ae4-4163-8b26-edc5ec2099e2\" (UID: \"6fbeac30-9ae4-4163-8b26-edc5ec2099e2\") " Oct 30 21:57:25.319932 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:25.319901 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fbeac30-9ae4-4163-8b26-edc5ec2099e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fbeac30-9ae4-4163-8b26-edc5ec2099e2-maas-api-openshift-ca\") pod \"6fbeac30-9ae4-4163-8b26-edc5ec2099e2\" (UID: \"6fbeac30-9ae4-4163-8b26-edc5ec2099e2\") " Oct 30 21:57:25.320298 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:25.320274 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fbeac30-9ae4-4163-8b26-edc5ec2099e2-maas-api-openshift-ca" pod "6fbeac30-9ae4-4163-8b26-edc5ec2099e2" (UID: "6fbeac30-9ae4-4163-8b26-edc5ec2099e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:25.321776 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:25.321757 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fbeac30-9ae4-4163-8b26-edc5ec2099e2-maas-api-xks-ca" pod "6fbeac30-9ae4-4163-8b26-edc5ec2099e2" (UID: "6fbeac30-9ae4-4163-8b26-edc5ec2099e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:25.321852 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:25.321780 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fbeac30-9ae4-4163-8b26-edc5ec2099e2-kube-api-access-fgl77" pod "6fbeac30-9ae4-4163-8b26-edc5ec2099e2" (UID: "6fbeac30-9ae4-4163-8b26-edc5ec2099e2"). InnerVolumeSpecName "kube-api-access-fgl77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:25.421198 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:25.421164 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fbeac30-9ae4-4163-8b26-edc5ec2099e2-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:25.421198 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:25.421194 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgl77\" (UniqueName: \"kubernetes.io/projected/6fbeac30-9ae4-4163-8b26-edc5ec2099e2-kube-api-access-fgl77\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:25.421198 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:25.421206 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fbeac30-9ae4-4163-8b26-edc5ec2099e2-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:26.158678 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:26.158652 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2aa268-wtnzv" Oct 30 21:57:26.158678 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:26.158662 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2aa268-wtnzv" event={"ID":"6fbeac30-9ae4-4163-8b26-edc5ec2099e2","Type":"ContainerDied","Data":"958bfd399c863caba8a151119f796865f21e44f065cf476b6e3615067930a1b1"} Oct 30 21:57:26.158941 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:26.158710 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="958bfd399c863caba8a151119f796865f21e44f065cf476b6e3615067930a1b1" Oct 30 21:57:27.767787 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:27.767756 4560 patch_prober.go:29] interesting pod/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:57:27.768255 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:27.767808 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" podUID="408e0e40-4d43-4ef8-adb0-07f371a17dd8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:57:28.720765 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.720740 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" Oct 30 21:57:28.764994 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.764971 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" Oct 30 21:57:28.782874 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.782843 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-istiod-ca-cert\") pod \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\" (UID: \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\") " Oct 30 21:57:28.783241 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.782907 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-istio-token\" (UniqueName: \"kubernetes.io/projected/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-istio-token\") pod \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\" (UID: \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\") " Oct 30 21:57:28.783241 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.782973 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0802666-5926-4a4f-a438-8e932494758c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0802666-5926-4a4f-a438-8e932494758c-workload-socket\") pod \"a0802666-5926-4a4f-a438-8e932494758c\" (UID: \"a0802666-5926-4a4f-a438-8e932494758c\") " Oct 30 21:57:28.783241 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.783011 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0802666-5926-4a4f-a438-8e932494758c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0802666-5926-4a4f-a438-8e932494758c-istio-podinfo\") pod \"a0802666-5926-4a4f-a438-8e932494758c\" (UID: \"a0802666-5926-4a4f-a438-8e932494758c\") " Oct 30 21:57:28.783241 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.783064 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0802666-5926-4a4f-a438-8e932494758c-kube-api-access-nkfvn\" (UniqueName: \"kubernetes.io/projected/a0802666-5926-4a4f-a438-8e932494758c-kube-api-access-nkfvn\") pod \"a0802666-5926-4a4f-a438-8e932494758c\" (UID: \"a0802666-5926-4a4f-a438-8e932494758c\") " Oct 30 21:57:28.783241 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.783097 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0802666-5926-4a4f-a438-8e932494758c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0802666-5926-4a4f-a438-8e932494758c-istio-data\") pod \"a0802666-5926-4a4f-a438-8e932494758c\" (UID: \"a0802666-5926-4a4f-a438-8e932494758c\") " Oct 30 21:57:28.783241 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.783141 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0802666-5926-4a4f-a438-8e932494758c-istio-token\" (UniqueName: \"kubernetes.io/projected/a0802666-5926-4a4f-a438-8e932494758c-istio-token\") pod \"a0802666-5926-4a4f-a438-8e932494758c\" (UID: \"a0802666-5926-4a4f-a438-8e932494758c\") " Oct 30 21:57:28.783241 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.783189 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0802666-5926-4a4f-a438-8e932494758c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0802666-5926-4a4f-a438-8e932494758c-credential-socket\") pod \"a0802666-5926-4a4f-a438-8e932494758c\" (UID: \"a0802666-5926-4a4f-a438-8e932494758c\") " Oct 30 21:57:28.783673 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.783247 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0802666-5926-4a4f-a438-8e932494758c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0802666-5926-4a4f-a438-8e932494758c-istiod-ca-cert\") pod \"a0802666-5926-4a4f-a438-8e932494758c\" (UID: \"a0802666-5926-4a4f-a438-8e932494758c\") " Oct 30 21:57:28.783673 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.783261 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0802666-5926-4a4f-a438-8e932494758c-workload-socket" pod "a0802666-5926-4a4f-a438-8e932494758c" (UID: "a0802666-5926-4a4f-a438-8e932494758c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:28.783673 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.783282 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-istio-envoy\") pod \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\" (UID: \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\") " Oct 30 21:57:28.783673 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.783294 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-istiod-ca-cert" pod "aaf8da80-7e8e-40a3-8c88-33b2325ef2aa" (UID: "aaf8da80-7e8e-40a3-8c88-33b2325ef2aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:28.783673 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.783519 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0802666-5926-4a4f-a438-8e932494758c-credential-socket" pod "a0802666-5926-4a4f-a438-8e932494758c" (UID: "a0802666-5926-4a4f-a438-8e932494758c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:28.783673 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.783564 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0802666-5926-4a4f-a438-8e932494758c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0802666-5926-4a4f-a438-8e932494758c-istio-envoy\") pod \"a0802666-5926-4a4f-a438-8e932494758c\" (UID: \"a0802666-5926-4a4f-a438-8e932494758c\") " Oct 30 21:57:28.783673 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.783622 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0802666-5926-4a4f-a438-8e932494758c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0802666-5926-4a4f-a438-8e932494758c-workload-certs\") pod \"a0802666-5926-4a4f-a438-8e932494758c\" (UID: \"a0802666-5926-4a4f-a438-8e932494758c\") " Oct 30 21:57:28.784046 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.783687 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-workload-socket\") pod \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\" (UID: \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\") " Oct 30 21:57:28.784046 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.783688 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0802666-5926-4a4f-a438-8e932494758c-istio-data" pod "a0802666-5926-4a4f-a438-8e932494758c" (UID: "a0802666-5926-4a4f-a438-8e932494758c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:28.784046 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.783763 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0802666-5926-4a4f-a438-8e932494758c-istiod-ca-cert" pod "a0802666-5926-4a4f-a438-8e932494758c" (UID: "a0802666-5926-4a4f-a438-8e932494758c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:28.784046 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.783770 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-istio-data\") pod \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\" (UID: \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\") " Oct 30 21:57:28.784046 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.783807 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-workload-certs\") pod \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\" (UID: \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\") " Oct 30 21:57:28.784046 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.783854 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-istio-podinfo\") pod \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\" (UID: \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\") " Oct 30 21:57:28.784046 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.783902 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-credential-socket\") pod \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\" (UID: \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\") " Oct 30 21:57:28.784046 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.783935 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-kube-api-access-2nhfh\" (UniqueName: \"kubernetes.io/projected/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-kube-api-access-2nhfh\") pod \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\" (UID: \"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa\") " Oct 30 21:57:28.784046 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.784007 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0802666-5926-4a4f-a438-8e932494758c-workload-certs" pod "a0802666-5926-4a4f-a438-8e932494758c" (UID: "a0802666-5926-4a4f-a438-8e932494758c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:28.784556 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.784180 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-workload-certs" pod "aaf8da80-7e8e-40a3-8c88-33b2325ef2aa" (UID: "aaf8da80-7e8e-40a3-8c88-33b2325ef2aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:28.784556 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.784335 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-workload-socket" pod "aaf8da80-7e8e-40a3-8c88-33b2325ef2aa" (UID: "aaf8da80-7e8e-40a3-8c88-33b2325ef2aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:28.784794 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.784763 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-istio-data" pod "aaf8da80-7e8e-40a3-8c88-33b2325ef2aa" (UID: "aaf8da80-7e8e-40a3-8c88-33b2325ef2aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:28.784794 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.784772 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-workload-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:28.784794 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.784793 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-istiod-ca-cert\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:28.785062 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.784807 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0802666-5926-4a4f-a438-8e932494758c-workload-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:28.785062 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.784819 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0802666-5926-4a4f-a438-8e932494758c-istio-data\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:28.785062 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.784832 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0802666-5926-4a4f-a438-8e932494758c-credential-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:28.785062 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.784845 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0802666-5926-4a4f-a438-8e932494758c-istiod-ca-cert\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:28.785062 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.784857 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0802666-5926-4a4f-a438-8e932494758c-workload-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:28.785062 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.784870 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-workload-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:28.785062 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.784894 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-credential-socket" pod "aaf8da80-7e8e-40a3-8c88-33b2325ef2aa" (UID: "aaf8da80-7e8e-40a3-8c88-33b2325ef2aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:28.785693 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.785667 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-istio-token" pod "aaf8da80-7e8e-40a3-8c88-33b2325ef2aa" (UID: "aaf8da80-7e8e-40a3-8c88-33b2325ef2aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:28.785783 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.785754 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0802666-5926-4a4f-a438-8e932494758c-istio-podinfo" pod "a0802666-5926-4a4f-a438-8e932494758c" (UID: "a0802666-5926-4a4f-a438-8e932494758c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:57:28.786091 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.786067 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-istio-envoy" pod "aaf8da80-7e8e-40a3-8c88-33b2325ef2aa" (UID: "aaf8da80-7e8e-40a3-8c88-33b2325ef2aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:28.786257 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.786230 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-istio-podinfo" pod "aaf8da80-7e8e-40a3-8c88-33b2325ef2aa" (UID: "aaf8da80-7e8e-40a3-8c88-33b2325ef2aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:57:28.786647 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.786630 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0802666-5926-4a4f-a438-8e932494758c-istio-token" pod "a0802666-5926-4a4f-a438-8e932494758c" (UID: "a0802666-5926-4a4f-a438-8e932494758c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:28.786988 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.786965 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-kube-api-access-2nhfh" pod "aaf8da80-7e8e-40a3-8c88-33b2325ef2aa" (UID: "aaf8da80-7e8e-40a3-8c88-33b2325ef2aa"). InnerVolumeSpecName "kube-api-access-2nhfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:28.787119 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.787106 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0802666-5926-4a4f-a438-8e932494758c-kube-api-access-nkfvn" pod "a0802666-5926-4a4f-a438-8e932494758c" (UID: "a0802666-5926-4a4f-a438-8e932494758c"). InnerVolumeSpecName "kube-api-access-nkfvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:28.787161 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.787127 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0802666-5926-4a4f-a438-8e932494758c-istio-envoy" pod "a0802666-5926-4a4f-a438-8e932494758c" (UID: "a0802666-5926-4a4f-a438-8e932494758c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:28.886169 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.886130 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-istio-podinfo\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:28.886169 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.886168 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-credential-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:28.886398 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.886184 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nhfh\" (UniqueName: \"kubernetes.io/projected/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-kube-api-access-2nhfh\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:28.886398 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.886199 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-istio-token\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:28.886398 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.886212 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0802666-5926-4a4f-a438-8e932494758c-istio-podinfo\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:28.886398 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.886227 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkfvn\" (UniqueName: \"kubernetes.io/projected/a0802666-5926-4a4f-a438-8e932494758c-kube-api-access-nkfvn\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:28.886398 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.886240 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0802666-5926-4a4f-a438-8e932494758c-istio-token\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:28.886398 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.886253 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-istio-envoy\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:28.886398 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.886266 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0802666-5926-4a4f-a438-8e932494758c-istio-envoy\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:28.886398 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:28.886280 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa-istio-data\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:29.177585 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:29.177548 4560 generic.go:356] "Generic (PLEG): container finished" podID="aaf8da80-7e8e-40a3-8c88-33b2325ef2aa" containerID="dc97903049a2d73dc0d3d2931402c73118f0019cf5f6396076ebeedd6f969433" exitCode=0 Oct 30 21:57:29.177753 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:29.177590 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" Oct 30 21:57:29.177753 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:29.177622 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" event={"ID":"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa","Type":"ContainerDied","Data":"dc97903049a2d73dc0d3d2931402c73118f0019cf5f6396076ebeedd6f969433"} Oct 30 21:57:29.177753 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:29.177673 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb" event={"ID":"aaf8da80-7e8e-40a3-8c88-33b2325ef2aa","Type":"ContainerDied","Data":"ac8118fc948ca453e4eedc572bb5a7dd8e39d57ee143d276a657fcee052adddd"} Oct 30 21:57:29.177753 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:29.177696 4560 scope.go:122] "RemoveContainer" containerID="dc97903049a2d73dc0d3d2931402c73118f0019cf5f6396076ebeedd6f969433" Oct 30 21:57:29.179038 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:29.179013 4560 generic.go:356] "Generic (PLEG): container finished" podID="a0802666-5926-4a4f-a438-8e932494758c" containerID="9e12acd38ecb1e266db2ec5f3db45fce431c0094c6e7044baa990d1f514cec6f" exitCode=0 Oct 30 21:57:29.179163 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:29.179043 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" event={"ID":"a0802666-5926-4a4f-a438-8e932494758c","Type":"ContainerDied","Data":"9e12acd38ecb1e266db2ec5f3db45fce431c0094c6e7044baa990d1f514cec6f"} Oct 30 21:57:29.179163 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:29.179064 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" Oct 30 21:57:29.179163 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:29.179092 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6" event={"ID":"a0802666-5926-4a4f-a438-8e932494758c","Type":"ContainerDied","Data":"87c124acef9d44ad4b251c3c37006c3870b74a5158f352cb8adabe1cd8af32a9"} Oct 30 21:57:29.189201 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:29.189119 4560 scope.go:122] "RemoveContainer" containerID="dc97903049a2d73dc0d3d2931402c73118f0019cf5f6396076ebeedd6f969433" Oct 30 21:57:29.189646 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:57:29.189625 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc97903049a2d73dc0d3d2931402c73118f0019cf5f6396076ebeedd6f969433\": container with ID starting with dc97903049a2d73dc0d3d2931402c73118f0019cf5f6396076ebeedd6f969433 not found: ID does not exist" containerID="dc97903049a2d73dc0d3d2931402c73118f0019cf5f6396076ebeedd6f969433" Oct 30 21:57:29.189738 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:29.189656 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc97903049a2d73dc0d3d2931402c73118f0019cf5f6396076ebeedd6f969433"} err="failed to get container status \"dc97903049a2d73dc0d3d2931402c73118f0019cf5f6396076ebeedd6f969433\": rpc error: code = NotFound desc = could not find container \"dc97903049a2d73dc0d3d2931402c73118f0019cf5f6396076ebeedd6f969433\": container with ID starting with dc97903049a2d73dc0d3d2931402c73118f0019cf5f6396076ebeedd6f969433 not found: ID does not exist" Oct 30 21:57:29.189738 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:29.189679 4560 scope.go:122] "RemoveContainer" containerID="9e12acd38ecb1e266db2ec5f3db45fce431c0094c6e7044baa990d1f514cec6f" Oct 30 21:57:29.200541 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:29.200512 4560 scope.go:122] "RemoveContainer" containerID="9e12acd38ecb1e266db2ec5f3db45fce431c0094c6e7044baa990d1f514cec6f" Oct 30 21:57:29.200839 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:57:29.200820 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e12acd38ecb1e266db2ec5f3db45fce431c0094c6e7044baa990d1f514cec6f\": container with ID starting with 9e12acd38ecb1e266db2ec5f3db45fce431c0094c6e7044baa990d1f514cec6f not found: ID does not exist" containerID="9e12acd38ecb1e266db2ec5f3db45fce431c0094c6e7044baa990d1f514cec6f" Oct 30 21:57:29.200904 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:29.200849 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e12acd38ecb1e266db2ec5f3db45fce431c0094c6e7044baa990d1f514cec6f"} err="failed to get container status \"9e12acd38ecb1e266db2ec5f3db45fce431c0094c6e7044baa990d1f514cec6f\": rpc error: code = NotFound desc = could not find container \"9e12acd38ecb1e266db2ec5f3db45fce431c0094c6e7044baa990d1f514cec6f\": container with ID starting with 9e12acd38ecb1e266db2ec5f3db45fce431c0094c6e7044baa990d1f514cec6f not found: ID does not exist" Oct 30 21:57:29.339431 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:29.339393 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb"] Oct 30 21:57:29.342777 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:29.342751 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-rkrzb"] Oct 30 21:57:29.446946 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:29.446896 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6"] Oct 30 21:57:29.453475 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:29.453449 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-4nnm6"] Oct 30 21:57:30.186878 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:30.186845 4560 generic.go:356] "Generic (PLEG): container finished" podID="408e0e40-4d43-4ef8-adb0-07f371a17dd8" containerID="e5fe40808d042be41381cb486e4fea21666d6ebdd287e51b10a4db08c289efb7" exitCode=0 Oct 30 21:57:30.187251 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:30.186914 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" event={"ID":"408e0e40-4d43-4ef8-adb0-07f371a17dd8","Type":"ContainerDied","Data":"e5fe40808d042be41381cb486e4fea21666d6ebdd287e51b10a4db08c289efb7"} Oct 30 21:57:30.327458 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:30.327426 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" Oct 30 21:57:30.417255 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:30.417184 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408e0e40-4d43-4ef8-adb0-07f371a17dd8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408e0e40-4d43-4ef8-adb0-07f371a17dd8-istio-envoy\") pod \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\" (UID: \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\") " Oct 30 21:57:30.417421 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:30.417284 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408e0e40-4d43-4ef8-adb0-07f371a17dd8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408e0e40-4d43-4ef8-adb0-07f371a17dd8-workload-socket\") pod \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\" (UID: \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\") " Oct 30 21:57:30.417421 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:30.417367 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/408e0e40-4d43-4ef8-adb0-07f371a17dd8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408e0e40-4d43-4ef8-adb0-07f371a17dd8-istio-podinfo\") pod \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\" (UID: \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\") " Oct 30 21:57:30.417531 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:30.417419 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408e0e40-4d43-4ef8-adb0-07f371a17dd8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408e0e40-4d43-4ef8-adb0-07f371a17dd8-istio-data\") pod \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\" (UID: \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\") " Oct 30 21:57:30.417531 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:30.417457 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408e0e40-4d43-4ef8-adb0-07f371a17dd8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408e0e40-4d43-4ef8-adb0-07f371a17dd8-credential-socket\") pod \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\" (UID: \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\") " Oct 30 21:57:30.417531 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:30.417522 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/408e0e40-4d43-4ef8-adb0-07f371a17dd8-istio-token\" (UniqueName: \"kubernetes.io/projected/408e0e40-4d43-4ef8-adb0-07f371a17dd8-istio-token\") pod \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\" (UID: \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\") " Oct 30 21:57:30.417683 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:30.417612 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408e0e40-4d43-4ef8-adb0-07f371a17dd8-workload-socket" pod "408e0e40-4d43-4ef8-adb0-07f371a17dd8" (UID: "408e0e40-4d43-4ef8-adb0-07f371a17dd8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:30.417683 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:30.417669 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408e0e40-4d43-4ef8-adb0-07f371a17dd8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408e0e40-4d43-4ef8-adb0-07f371a17dd8-workload-certs\") pod \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\" (UID: \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\") " Oct 30 21:57:30.417790 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:30.417728 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/408e0e40-4d43-4ef8-adb0-07f371a17dd8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408e0e40-4d43-4ef8-adb0-07f371a17dd8-istiod-ca-cert\") pod \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\" (UID: \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\") " Oct 30 21:57:30.417790 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:30.417739 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408e0e40-4d43-4ef8-adb0-07f371a17dd8-credential-socket" pod "408e0e40-4d43-4ef8-adb0-07f371a17dd8" (UID: "408e0e40-4d43-4ef8-adb0-07f371a17dd8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:30.417790 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:30.417755 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/408e0e40-4d43-4ef8-adb0-07f371a17dd8-kube-api-access-dvfb5\" (UniqueName: \"kubernetes.io/projected/408e0e40-4d43-4ef8-adb0-07f371a17dd8-kube-api-access-dvfb5\") pod \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\" (UID: \"408e0e40-4d43-4ef8-adb0-07f371a17dd8\") " Oct 30 21:57:30.417790 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:30.417750 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408e0e40-4d43-4ef8-adb0-07f371a17dd8-istio-data" pod "408e0e40-4d43-4ef8-adb0-07f371a17dd8" (UID: "408e0e40-4d43-4ef8-adb0-07f371a17dd8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:30.417972 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:30.417920 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408e0e40-4d43-4ef8-adb0-07f371a17dd8-workload-certs" pod "408e0e40-4d43-4ef8-adb0-07f371a17dd8" (UID: "408e0e40-4d43-4ef8-adb0-07f371a17dd8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:30.418114 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:30.418080 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/408e0e40-4d43-4ef8-adb0-07f371a17dd8-istiod-ca-cert" pod "408e0e40-4d43-4ef8-adb0-07f371a17dd8" (UID: "408e0e40-4d43-4ef8-adb0-07f371a17dd8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:30.418365 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:30.418322 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408e0e40-4d43-4ef8-adb0-07f371a17dd8-workload-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:30.418476 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:30.418368 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408e0e40-4d43-4ef8-adb0-07f371a17dd8-istiod-ca-cert\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:30.418476 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:30.418383 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408e0e40-4d43-4ef8-adb0-07f371a17dd8-workload-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:30.418476 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:30.418396 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408e0e40-4d43-4ef8-adb0-07f371a17dd8-istio-data\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:30.418476 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:30.418410 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408e0e40-4d43-4ef8-adb0-07f371a17dd8-credential-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:30.419804 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:30.419780 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/408e0e40-4d43-4ef8-adb0-07f371a17dd8-istio-podinfo" pod "408e0e40-4d43-4ef8-adb0-07f371a17dd8" (UID: "408e0e40-4d43-4ef8-adb0-07f371a17dd8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:57:30.419895 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:30.419832 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/408e0e40-4d43-4ef8-adb0-07f371a17dd8-kube-api-access-dvfb5" pod "408e0e40-4d43-4ef8-adb0-07f371a17dd8" (UID: "408e0e40-4d43-4ef8-adb0-07f371a17dd8"). InnerVolumeSpecName "kube-api-access-dvfb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:30.419944 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:30.419902 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/408e0e40-4d43-4ef8-adb0-07f371a17dd8-istio-token" pod "408e0e40-4d43-4ef8-adb0-07f371a17dd8" (UID: "408e0e40-4d43-4ef8-adb0-07f371a17dd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:30.419944 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:30.419910 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408e0e40-4d43-4ef8-adb0-07f371a17dd8-istio-envoy" pod "408e0e40-4d43-4ef8-adb0-07f371a17dd8" (UID: "408e0e40-4d43-4ef8-adb0-07f371a17dd8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:30.520015 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:30.519987 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408e0e40-4d43-4ef8-adb0-07f371a17dd8-istio-token\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:30.520015 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:30.520014 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvfb5\" (UniqueName: \"kubernetes.io/projected/408e0e40-4d43-4ef8-adb0-07f371a17dd8-kube-api-access-dvfb5\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:30.520015 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:30.520023 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408e0e40-4d43-4ef8-adb0-07f371a17dd8-istio-envoy\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:30.520267 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:30.520033 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408e0e40-4d43-4ef8-adb0-07f371a17dd8-istio-podinfo\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:30.584856 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:30.584813 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0802666-5926-4a4f-a438-8e932494758c" path="/var/lib/kubelet/pods/a0802666-5926-4a4f-a438-8e932494758c/volumes" Oct 30 21:57:30.585381 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:30.585366 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaf8da80-7e8e-40a3-8c88-33b2325ef2aa" path="/var/lib/kubelet/pods/aaf8da80-7e8e-40a3-8c88-33b2325ef2aa/volumes" Oct 30 21:57:31.197695 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:31.197663 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" Oct 30 21:57:31.197695 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:31.197688 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j" event={"ID":"408e0e40-4d43-4ef8-adb0-07f371a17dd8","Type":"ContainerDied","Data":"9046fbab9ad3e46df2a80271619df423704f0c5d194f5bcbfc310e349c14be32"} Oct 30 21:57:31.198181 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:31.197749 4560 scope.go:122] "RemoveContainer" containerID="e5fe40808d042be41381cb486e4fea21666d6ebdd287e51b10a4db08c289efb7" Oct 30 21:57:31.373407 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:31.373375 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j"] Oct 30 21:57:31.419866 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:31.419833 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-6wr5j"] Oct 30 21:57:32.576705 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:32.576676 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="408e0e40-4d43-4ef8-adb0-07f371a17dd8" path="/var/lib/kubelet/pods/408e0e40-4d43-4ef8-adb0-07f371a17dd8/volumes" Oct 30 21:57:35.872754 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:35.872716 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:57:47.089313 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:47.089276 4560 scope.go:122] "RemoveContainer" containerID="dcd7e38bb903266af81cb601f4ebe51a2134ebe599279be937de978977efd42c" Oct 30 21:57:47.097879 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:47.097858 4560 scope.go:122] "RemoveContainer" containerID="e78ed765ce8e17926acb42d69efaaeceee1012b83830837477751f62b1786c91" Oct 30 21:57:47.105773 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:47.105752 4560 scope.go:122] "RemoveContainer" containerID="956e10023d9401776ea08ed95e448b1fb5cfdd97098e6c7bd4ee7ba0fefec90f" Oct 30 21:57:49.686211 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:57:49.686160 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:57:51.740450 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:51.740418 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-lln9s"] Oct 30 21:57:51.786539 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:51.786510 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-lln9s"] Oct 30 21:57:52.576665 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:57:52.576634 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9214c76a-32cc-45de-8130-a22060868346" path="/var/lib/kubelet/pods/9214c76a-32cc-45de-8130-a22060868346/volumes" Oct 30 21:58:06.668424 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:06.668283 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bfdf30b9-66766fc856-gscj4"] Oct 30 21:58:06.674625 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:06.669082 4560 cpu_manager.go:414] "containerMap: removing container" podUID="408e0e40-4d43-4ef8-adb0-07f371a17dd8" containerName="istio-proxy" Oct 30 21:58:06.674625 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:06.669106 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="408e0e40-4d43-4ef8-adb0-07f371a17dd8" containerName="istio-proxy" Oct 30 21:58:06.674625 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:06.669139 4560 cpu_manager.go:414] "containerMap: removing container" podUID="6fbeac30-9ae4-4163-8b26-edc5ec2099e2" containerName="revoke-keys" Oct 30 21:58:06.674625 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:06.669147 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fbeac30-9ae4-4163-8b26-edc5ec2099e2" containerName="revoke-keys" Oct 30 21:58:06.674625 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:06.669168 4560 cpu_manager.go:414] "containerMap: removing container" podUID="a0802666-5926-4a4f-a438-8e932494758c" containerName="istio-proxy" Oct 30 21:58:06.674625 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:06.669178 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0802666-5926-4a4f-a438-8e932494758c" containerName="istio-proxy" Oct 30 21:58:06.674625 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:06.669199 4560 cpu_manager.go:414] "containerMap: removing container" podUID="aaf8da80-7e8e-40a3-8c88-33b2325ef2aa" containerName="istio-proxy" Oct 30 21:58:06.674625 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:06.669205 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaf8da80-7e8e-40a3-8c88-33b2325ef2aa" containerName="istio-proxy" Oct 30 21:58:06.674625 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:06.669333 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="408e0e40-4d43-4ef8-adb0-07f371a17dd8" containerName="istio-proxy" Oct 30 21:58:06.674625 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:06.669370 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fbeac30-9ae4-4163-8b26-edc5ec2099e2" containerName="revoke-keys" Oct 30 21:58:06.674625 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:06.669382 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0802666-5926-4a4f-a438-8e932494758c" containerName="istio-proxy" Oct 30 21:58:06.674625 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:06.669394 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaf8da80-7e8e-40a3-8c88-33b2325ef2aa" containerName="istio-proxy" Oct 30 21:58:06.698949 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:06.698918 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bfdf30b9-66766fc856-gscj4" Oct 30 21:58:06.746447 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:06.746411 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bfdf30b9-66766fc856-gscj4"] Oct 30 21:58:06.817472 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:06.817420 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bfdf30b9-6c99dcd5f5-sp9kv"] Oct 30 21:58:06.822904 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:06.822122 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bfdf30b9-6c99dcd5f5-sp9kv" Oct 30 21:58:06.854618 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:06.851175 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45zsx\" (UniqueName: \"kubernetes.io/projected/69c4f942-a6f7-4d0f-9f30-10a1de67e587-kube-api-access-45zsx\") pod \"payload-processing-e2e-mt-bfdf30b9-66766fc856-gscj4\" (UID: \"69c4f942-a6f7-4d0f-9f30-10a1de67e587\") " pod="openshift-ingress/payload-processing-e2e-mt-bfdf30b9-66766fc856-gscj4" Oct 30 21:58:06.854618 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:06.851290 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69c4f942-a6f7-4d0f-9f30-10a1de67e587-plugins-config-volume\") pod \"payload-processing-e2e-mt-bfdf30b9-66766fc856-gscj4\" (UID: \"69c4f942-a6f7-4d0f-9f30-10a1de67e587\") " pod="openshift-ingress/payload-processing-e2e-mt-bfdf30b9-66766fc856-gscj4" Oct 30 21:58:06.854618 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:06.852690 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-bfdf30b9-dockercfg-vk57n\"" Oct 30 21:58:06.854618 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:06.852781 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-bfdf30b9\"" Oct 30 21:58:06.886945 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:06.886879 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bfdf30b9-6c99dcd5f5-sp9kv"] Oct 30 21:58:06.954416 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:06.954137 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69c4f942-a6f7-4d0f-9f30-10a1de67e587-plugins-config-volume\") pod \"payload-processing-e2e-mt-bfdf30b9-66766fc856-gscj4\" (UID: \"69c4f942-a6f7-4d0f-9f30-10a1de67e587\") " pod="openshift-ingress/payload-processing-e2e-mt-bfdf30b9-66766fc856-gscj4" Oct 30 21:58:06.954416 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:06.954226 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q9n5\" (UniqueName: \"kubernetes.io/projected/c6372f6e-8f23-43b4-8fbf-33f0fb5f61ac-kube-api-access-2q9n5\") pod \"payload-pre-processing-e2e-mt-bfdf30b9-6c99dcd5f5-sp9kv\" (UID: \"c6372f6e-8f23-43b4-8fbf-33f0fb5f61ac\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bfdf30b9-6c99dcd5f5-sp9kv" Oct 30 21:58:06.954416 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:06.954363 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6372f6e-8f23-43b4-8fbf-33f0fb5f61ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bfdf30b9-6c99dcd5f5-sp9kv\" (UID: \"c6372f6e-8f23-43b4-8fbf-33f0fb5f61ac\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bfdf30b9-6c99dcd5f5-sp9kv" Oct 30 21:58:06.954731 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:06.954422 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45zsx\" (UniqueName: \"kubernetes.io/projected/69c4f942-a6f7-4d0f-9f30-10a1de67e587-kube-api-access-45zsx\") pod \"payload-processing-e2e-mt-bfdf30b9-66766fc856-gscj4\" (UID: \"69c4f942-a6f7-4d0f-9f30-10a1de67e587\") " pod="openshift-ingress/payload-processing-e2e-mt-bfdf30b9-66766fc856-gscj4" Oct 30 21:58:06.955711 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:06.955357 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69c4f942-a6f7-4d0f-9f30-10a1de67e587-plugins-config-volume\") pod \"payload-processing-e2e-mt-bfdf30b9-66766fc856-gscj4\" (UID: \"69c4f942-a6f7-4d0f-9f30-10a1de67e587\") " pod="openshift-ingress/payload-processing-e2e-mt-bfdf30b9-66766fc856-gscj4" Oct 30 21:58:07.024627 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:07.024599 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45zsx\" (UniqueName: \"kubernetes.io/projected/69c4f942-a6f7-4d0f-9f30-10a1de67e587-kube-api-access-45zsx\") pod \"payload-processing-e2e-mt-bfdf30b9-66766fc856-gscj4\" (UID: \"69c4f942-a6f7-4d0f-9f30-10a1de67e587\") " pod="openshift-ingress/payload-processing-e2e-mt-bfdf30b9-66766fc856-gscj4" Oct 30 21:58:07.052215 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:07.052181 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bfdf30b9-66766fc856-gscj4" Oct 30 21:58:07.055787 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:07.055708 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6372f6e-8f23-43b4-8fbf-33f0fb5f61ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bfdf30b9-6c99dcd5f5-sp9kv\" (UID: \"c6372f6e-8f23-43b4-8fbf-33f0fb5f61ac\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bfdf30b9-6c99dcd5f5-sp9kv" Oct 30 21:58:07.057242 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:07.055861 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q9n5\" (UniqueName: \"kubernetes.io/projected/c6372f6e-8f23-43b4-8fbf-33f0fb5f61ac-kube-api-access-2q9n5\") pod \"payload-pre-processing-e2e-mt-bfdf30b9-6c99dcd5f5-sp9kv\" (UID: \"c6372f6e-8f23-43b4-8fbf-33f0fb5f61ac\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bfdf30b9-6c99dcd5f5-sp9kv" Oct 30 21:58:07.057242 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:07.057094 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6372f6e-8f23-43b4-8fbf-33f0fb5f61ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bfdf30b9-6c99dcd5f5-sp9kv\" (UID: \"c6372f6e-8f23-43b4-8fbf-33f0fb5f61ac\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bfdf30b9-6c99dcd5f5-sp9kv" Oct 30 21:58:07.118552 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:07.118491 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q9n5\" (UniqueName: \"kubernetes.io/projected/c6372f6e-8f23-43b4-8fbf-33f0fb5f61ac-kube-api-access-2q9n5\") pod \"payload-pre-processing-e2e-mt-bfdf30b9-6c99dcd5f5-sp9kv\" (UID: \"c6372f6e-8f23-43b4-8fbf-33f0fb5f61ac\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bfdf30b9-6c99dcd5f5-sp9kv" Oct 30 21:58:07.171110 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:07.171031 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bfdf30b9-6c99dcd5f5-sp9kv" Oct 30 21:58:07.453231 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:07.453203 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bfdf30b9-66766fc856-gscj4"] Oct 30 21:58:07.454941 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:58:07.454904 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c4f942_a6f7_4d0f_9f30_10a1de67e587.slice/crio-a0da52d466950589a3d1c28d1717042a194a8b295a83dd5fee168437af1c98f6 WatchSource:0}: Error finding container a0da52d466950589a3d1c28d1717042a194a8b295a83dd5fee168437af1c98f6: Status 404 returned error can't find the container with id a0da52d466950589a3d1c28d1717042a194a8b295a83dd5fee168437af1c98f6 Oct 30 21:58:07.520276 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:07.520246 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bfdf30b9-66766fc856-gscj4" event={"ID":"69c4f942-a6f7-4d0f-9f30-10a1de67e587","Type":"ContainerStarted","Data":"a0da52d466950589a3d1c28d1717042a194a8b295a83dd5fee168437af1c98f6"} Oct 30 21:58:07.580284 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:07.580249 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bfdf30b9-6c99dcd5f5-sp9kv"] Oct 30 21:58:07.584961 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:58:07.584920 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6372f6e_8f23_43b4_8fbf_33f0fb5f61ac.slice/crio-1799db2f5faf5ec17c7b0a2466a3fa94d5e5e0b405dc15ef48c3437dae1565d4 WatchSource:0}: Error finding container 1799db2f5faf5ec17c7b0a2466a3fa94d5e5e0b405dc15ef48c3437dae1565d4: Status 404 returned error can't find the container with id 1799db2f5faf5ec17c7b0a2466a3fa94d5e5e0b405dc15ef48c3437dae1565d4 Oct 30 21:58:08.527121 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:08.527082 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bfdf30b9-6c99dcd5f5-sp9kv" event={"ID":"c6372f6e-8f23-43b4-8fbf-33f0fb5f61ac","Type":"ContainerStarted","Data":"0e2866238e4ababe75c029fa69d4bf0ffd3c778ab6b45644775f29138d52478d"} Oct 30 21:58:08.527603 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:08.527145 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bfdf30b9-6c99dcd5f5-sp9kv" event={"ID":"c6372f6e-8f23-43b4-8fbf-33f0fb5f61ac","Type":"ContainerStarted","Data":"1799db2f5faf5ec17c7b0a2466a3fa94d5e5e0b405dc15ef48c3437dae1565d4"} Oct 30 21:58:08.527603 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:08.527175 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-bfdf30b9-6c99dcd5f5-sp9kv" Oct 30 21:58:08.528754 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:08.528726 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bfdf30b9-66766fc856-gscj4" event={"ID":"69c4f942-a6f7-4d0f-9f30-10a1de67e587","Type":"ContainerStarted","Data":"d262adbd4f8258d45eb2c9a70adb0b526a9e9e2faf427d68cdd0f53b45afbf9f"} Oct 30 21:58:08.528866 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:08.528840 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-bfdf30b9-66766fc856-gscj4" Oct 30 21:58:08.639573 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:08.639522 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-bfdf30b9-6c99dcd5f5-sp9kv" podStartSLOduration=2.6395101370000003 podStartE2EDuration="2.639510137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:08.637272588 +0000 UTC m=+2664.191972365" watchObservedRunningTime="2026-10-30 21:58:08.639510137 +0000 UTC m=+2664.194209913" Oct 30 21:58:08.775153 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:08.773210 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-bfdf30b9-66766fc856-gscj4" podStartSLOduration=2.773187914 podStartE2EDuration="2.773187914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:08.760159196 +0000 UTC m=+2664.314858972" watchObservedRunningTime="2026-10-30 21:58:08.773187914 +0000 UTC m=+2664.327887692" Oct 30 21:58:10.098170 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:10.098139 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a02f114-r5qqf"] Oct 30 21:58:10.102072 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:10.102054 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a02f114-r5qqf" Oct 30 21:58:10.155439 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:10.155409 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:58:10.202247 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:10.202210 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:58:10.202498 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:10.202288 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:58:10.202498 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:10.202403 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:58:10.205132 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:10.205104 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae6a851b-aa85-4993-a7d9-134b91a6ad05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a02f114-r5qqf\" (UID: \"ae6a851b-aa85-4993-a7d9-134b91a6ad05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a02f114-r5qqf" Oct 30 21:58:10.205257 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:10.205231 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae6a851b-aa85-4993-a7d9-134b91a6ad05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a02f114-r5qqf\" (UID: \"ae6a851b-aa85-4993-a7d9-134b91a6ad05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a02f114-r5qqf" Oct 30 21:58:10.205332 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:10.205295 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnr6d\" (UniqueName: \"kubernetes.io/projected/ae6a851b-aa85-4993-a7d9-134b91a6ad05-kube-api-access-rnr6d\") pod \"maas-api-revoke-sub-2a02f114-r5qqf\" (UID: \"ae6a851b-aa85-4993-a7d9-134b91a6ad05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a02f114-r5qqf" Oct 30 21:58:10.205736 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:10.205714 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a02f114-r5qqf"] Oct 30 21:58:10.306793 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:10.306762 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae6a851b-aa85-4993-a7d9-134b91a6ad05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a02f114-r5qqf\" (UID: \"ae6a851b-aa85-4993-a7d9-134b91a6ad05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a02f114-r5qqf" Oct 30 21:58:10.306986 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:10.306811 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnr6d\" (UniqueName: \"kubernetes.io/projected/ae6a851b-aa85-4993-a7d9-134b91a6ad05-kube-api-access-rnr6d\") pod \"maas-api-revoke-sub-2a02f114-r5qqf\" (UID: \"ae6a851b-aa85-4993-a7d9-134b91a6ad05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a02f114-r5qqf" Oct 30 21:58:10.306986 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:10.306950 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae6a851b-aa85-4993-a7d9-134b91a6ad05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a02f114-r5qqf\" (UID: \"ae6a851b-aa85-4993-a7d9-134b91a6ad05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a02f114-r5qqf" Oct 30 21:58:10.307607 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:10.307585 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae6a851b-aa85-4993-a7d9-134b91a6ad05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a02f114-r5qqf\" (UID: \"ae6a851b-aa85-4993-a7d9-134b91a6ad05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a02f114-r5qqf" Oct 30 21:58:10.309151 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:10.309134 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae6a851b-aa85-4993-a7d9-134b91a6ad05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a02f114-r5qqf\" (UID: \"ae6a851b-aa85-4993-a7d9-134b91a6ad05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a02f114-r5qqf" Oct 30 21:58:10.366231 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:10.366171 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnr6d\" (UniqueName: \"kubernetes.io/projected/ae6a851b-aa85-4993-a7d9-134b91a6ad05-kube-api-access-rnr6d\") pod \"maas-api-revoke-sub-2a02f114-r5qqf\" (UID: \"ae6a851b-aa85-4993-a7d9-134b91a6ad05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a02f114-r5qqf" Oct 30 21:58:10.417115 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:10.417088 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a02f114-r5qqf" Oct 30 21:58:10.797591 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:10.797560 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a02f114-r5qqf"] Oct 30 21:58:10.802207 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:58:10.802180 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae6a851b_aa85_4993_a7d9_134b91a6ad05.slice/crio-48a5d54988d837213322705435b14b9caf523d061e407e9e5eaa5811df3fcb79 WatchSource:0}: Error finding container 48a5d54988d837213322705435b14b9caf523d061e407e9e5eaa5811df3fcb79: Status 404 returned error can't find the container with id 48a5d54988d837213322705435b14b9caf523d061e407e9e5eaa5811df3fcb79 Oct 30 21:58:11.548023 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:11.547990 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a02f114-r5qqf" event={"ID":"ae6a851b-aa85-4993-a7d9-134b91a6ad05","Type":"ContainerStarted","Data":"a0c831c6dd934318d9501c1a9fb0a39313872fa2b124d667662c06a09828b624"} Oct 30 21:58:11.548494 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:11.548048 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a02f114-r5qqf" event={"ID":"ae6a851b-aa85-4993-a7d9-134b91a6ad05","Type":"ContainerStarted","Data":"48a5d54988d837213322705435b14b9caf523d061e407e9e5eaa5811df3fcb79"} Oct 30 21:58:11.549616 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:11.549591 4560 generic.go:356] "Generic (PLEG): container finished" podID="64e8e251-1ae0-42cc-b46b-e8f11b3395ff" containerID="601da35e07d40b926692b97ec4f18d9ce1169c75d199608c6fb69fb27769b567" exitCode=0 Oct 30 21:58:11.549715 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:11.549641 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" event={"ID":"64e8e251-1ae0-42cc-b46b-e8f11b3395ff","Type":"ContainerDied","Data":"601da35e07d40b926692b97ec4f18d9ce1169c75d199608c6fb69fb27769b567"} Oct 30 21:58:11.549970 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:11.549956 4560 scope.go:122] "RemoveContainer" containerID="601da35e07d40b926692b97ec4f18d9ce1169c75d199608c6fb69fb27769b567" Oct 30 21:58:11.551364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:11.551323 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:58:11.551426 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:11.551408 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:58:11.551462 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:11.551443 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:58:11.774950 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:11.774901 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a02f114-r5qqf" podStartSLOduration=1.774887189 podStartE2EDuration="1.774887189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:11.659232066 +0000 UTC m=+2667.213931842" watchObservedRunningTime="2026-10-30 21:58:11.774887189 +0000 UTC m=+2667.329586965" Oct 30 21:58:12.559853 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:12.559817 4560 generic.go:356] "Generic (PLEG): container finished" podID="ae6a851b-aa85-4993-a7d9-134b91a6ad05" containerID="a0c831c6dd934318d9501c1a9fb0a39313872fa2b124d667662c06a09828b624" exitCode=0 Oct 30 21:58:12.560321 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:12.559905 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a02f114-r5qqf" event={"ID":"ae6a851b-aa85-4993-a7d9-134b91a6ad05","Type":"ContainerDied","Data":"a0c831c6dd934318d9501c1a9fb0a39313872fa2b124d667662c06a09828b624"} Oct 30 21:58:12.561993 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:12.561971 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" event={"ID":"64e8e251-1ae0-42cc-b46b-e8f11b3395ff","Type":"ContainerStarted","Data":"d76a6c9fed686429cada32d362b582bfc579ba7fa56470957969ae5570e971cb"} Oct 30 21:58:13.559671 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:13.559637 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" Oct 30 21:58:13.561110 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:13.561081 4560 patch_prober.go:29] interesting pod/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 21:58:13.561565 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:13.561533 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" podUID="64e8e251-1ae0-42cc-b46b-e8f11b3395ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 21:58:13.758488 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:13.758461 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a02f114-r5qqf" Oct 30 21:58:13.864634 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:13.864553 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae6a851b-aa85-4993-a7d9-134b91a6ad05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae6a851b-aa85-4993-a7d9-134b91a6ad05-maas-api-xks-ca\") pod \"ae6a851b-aa85-4993-a7d9-134b91a6ad05\" (UID: \"ae6a851b-aa85-4993-a7d9-134b91a6ad05\") " Oct 30 21:58:13.864634 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:13.864601 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae6a851b-aa85-4993-a7d9-134b91a6ad05-kube-api-access-rnr6d\" (UniqueName: \"kubernetes.io/projected/ae6a851b-aa85-4993-a7d9-134b91a6ad05-kube-api-access-rnr6d\") pod \"ae6a851b-aa85-4993-a7d9-134b91a6ad05\" (UID: \"ae6a851b-aa85-4993-a7d9-134b91a6ad05\") " Oct 30 21:58:13.864825 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:13.864638 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae6a851b-aa85-4993-a7d9-134b91a6ad05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae6a851b-aa85-4993-a7d9-134b91a6ad05-maas-api-openshift-ca\") pod \"ae6a851b-aa85-4993-a7d9-134b91a6ad05\" (UID: \"ae6a851b-aa85-4993-a7d9-134b91a6ad05\") " Oct 30 21:58:13.865233 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:13.865209 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae6a851b-aa85-4993-a7d9-134b91a6ad05-maas-api-openshift-ca" pod "ae6a851b-aa85-4993-a7d9-134b91a6ad05" (UID: "ae6a851b-aa85-4993-a7d9-134b91a6ad05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:13.866865 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:13.866818 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae6a851b-aa85-4993-a7d9-134b91a6ad05-kube-api-access-rnr6d" pod "ae6a851b-aa85-4993-a7d9-134b91a6ad05" (UID: "ae6a851b-aa85-4993-a7d9-134b91a6ad05"). InnerVolumeSpecName "kube-api-access-rnr6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:13.866964 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:13.866875 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae6a851b-aa85-4993-a7d9-134b91a6ad05-maas-api-xks-ca" pod "ae6a851b-aa85-4993-a7d9-134b91a6ad05" (UID: "ae6a851b-aa85-4993-a7d9-134b91a6ad05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:13.966166 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:13.966134 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnr6d\" (UniqueName: \"kubernetes.io/projected/ae6a851b-aa85-4993-a7d9-134b91a6ad05-kube-api-access-rnr6d\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:13.966166 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:13.966168 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae6a851b-aa85-4993-a7d9-134b91a6ad05-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:13.966368 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:13.966184 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae6a851b-aa85-4993-a7d9-134b91a6ad05-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:14.308292 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:14.308259 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" Oct 30 21:58:14.559193 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:14.559111 4560 patch_prober.go:29] interesting pod/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 21:58:14.559193 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:14.559164 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" podUID="64e8e251-1ae0-42cc-b46b-e8f11b3395ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 21:58:14.578389 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:14.578362 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a02f114-r5qqf" Oct 30 21:58:14.579749 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:14.579721 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a02f114-r5qqf" event={"ID":"ae6a851b-aa85-4993-a7d9-134b91a6ad05","Type":"ContainerDied","Data":"48a5d54988d837213322705435b14b9caf523d061e407e9e5eaa5811df3fcb79"} Oct 30 21:58:14.579832 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:14.579772 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48a5d54988d837213322705435b14b9caf523d061e407e9e5eaa5811df3fcb79" Oct 30 21:58:15.559114 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:15.559072 4560 patch_prober.go:29] interesting pod/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 21:58:15.559295 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:15.559128 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" podUID="64e8e251-1ae0-42cc-b46b-e8f11b3395ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 21:58:16.559056 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:16.559023 4560 patch_prober.go:29] interesting pod/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 21:58:16.559482 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:16.559074 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" podUID="64e8e251-1ae0-42cc-b46b-e8f11b3395ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 21:58:17.563674 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:17.563639 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" Oct 30 21:58:17.564726 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:17.564706 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" Oct 30 21:58:19.537786 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:19.537754 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-bfdf30b9-66766fc856-gscj4" Oct 30 21:58:19.538213 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:19.537823 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-bfdf30b9-6c99dcd5f5-sp9kv" Oct 30 21:58:19.688950 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:58:19.688910 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:58:22.425464 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:22.425434 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7d7f57-c24mb"] Oct 30 21:58:22.471993 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:22.471967 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7d7f57-c24mb"] Oct 30 21:58:22.575449 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:22.575418 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac521211-4a30-4704-832d-367e16899231" path="/var/lib/kubelet/pods/ac521211-4a30-4704-832d-367e16899231/volumes" Oct 30 21:58:28.541160 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:28.541125 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:58:28.976922 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:28.976895 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2aa268-wtnzv"] Oct 30 21:58:29.023405 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:29.023373 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2aa268-wtnzv"] Oct 30 21:58:30.576653 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:30.576613 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fbeac30-9ae4-4163-8b26-edc5ec2099e2" path="/var/lib/kubelet/pods/6fbeac30-9ae4-4163-8b26-edc5ec2099e2/volumes" Oct 30 21:58:33.905732 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:33.905688 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-554f46-trdg6"] Oct 30 21:58:33.906397 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:33.906375 4560 cpu_manager.go:414] "containerMap: removing container" podUID="ae6a851b-aa85-4993-a7d9-134b91a6ad05" containerName="revoke-keys" Oct 30 21:58:33.906511 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:33.906397 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae6a851b-aa85-4993-a7d9-134b91a6ad05" containerName="revoke-keys" Oct 30 21:58:33.906568 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:33.906551 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae6a851b-aa85-4993-a7d9-134b91a6ad05" containerName="revoke-keys" Oct 30 21:58:33.911605 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:33.911584 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-554f46-trdg6" Oct 30 21:58:34.031549 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:34.031501 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l86w\" (UniqueName: \"kubernetes.io/projected/a42c755a-8283-4f86-86f4-9bdbe1753bdd-kube-api-access-2l86w\") pod \"maas-api-revoke-keys-e2e-worker-w4-554f46-trdg6\" (UID: \"a42c755a-8283-4f86-86f4-9bdbe1753bdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-554f46-trdg6" Oct 30 21:58:34.031848 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:34.031824 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a42c755a-8283-4f86-86f4-9bdbe1753bdd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-554f46-trdg6\" (UID: \"a42c755a-8283-4f86-86f4-9bdbe1753bdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-554f46-trdg6" Oct 30 21:58:34.032044 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:34.032024 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a42c755a-8283-4f86-86f4-9bdbe1753bdd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-554f46-trdg6\" (UID: \"a42c755a-8283-4f86-86f4-9bdbe1753bdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-554f46-trdg6" Oct 30 21:58:34.042468 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:34.042435 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:58:34.042639 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:34.042479 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:58:34.042639 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:34.042509 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:58:34.042639 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:34.042551 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:58:34.042796 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:34.042656 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-554f46-trdg6"] Oct 30 21:58:34.133811 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:34.133779 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l86w\" (UniqueName: \"kubernetes.io/projected/a42c755a-8283-4f86-86f4-9bdbe1753bdd-kube-api-access-2l86w\") pod \"maas-api-revoke-keys-e2e-worker-w4-554f46-trdg6\" (UID: \"a42c755a-8283-4f86-86f4-9bdbe1753bdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-554f46-trdg6" Oct 30 21:58:34.134005 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:34.133841 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a42c755a-8283-4f86-86f4-9bdbe1753bdd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-554f46-trdg6\" (UID: \"a42c755a-8283-4f86-86f4-9bdbe1753bdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-554f46-trdg6" Oct 30 21:58:34.134005 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:34.133940 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a42c755a-8283-4f86-86f4-9bdbe1753bdd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-554f46-trdg6\" (UID: \"a42c755a-8283-4f86-86f4-9bdbe1753bdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-554f46-trdg6" Oct 30 21:58:34.134530 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:34.134512 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a42c755a-8283-4f86-86f4-9bdbe1753bdd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-554f46-trdg6\" (UID: \"a42c755a-8283-4f86-86f4-9bdbe1753bdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-554f46-trdg6" Oct 30 21:58:34.136487 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:34.136461 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a42c755a-8283-4f86-86f4-9bdbe1753bdd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-554f46-trdg6\" (UID: \"a42c755a-8283-4f86-86f4-9bdbe1753bdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-554f46-trdg6" Oct 30 21:58:34.202492 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:34.202410 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l86w\" (UniqueName: \"kubernetes.io/projected/a42c755a-8283-4f86-86f4-9bdbe1753bdd-kube-api-access-2l86w\") pod \"maas-api-revoke-keys-e2e-worker-w4-554f46-trdg6\" (UID: \"a42c755a-8283-4f86-86f4-9bdbe1753bdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-554f46-trdg6" Oct 30 21:58:34.224658 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:34.224634 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-554f46-trdg6" Oct 30 21:58:34.605167 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:34.605123 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-554f46-trdg6"] Oct 30 21:58:34.611078 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:58:34.611051 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda42c755a_8283_4f86_86f4_9bdbe1753bdd.slice/crio-af5ef8732db464515ef4129bed83c2202c334123d808c56bc596c82010613148 WatchSource:0}: Error finding container af5ef8732db464515ef4129bed83c2202c334123d808c56bc596c82010613148: Status 404 returned error can't find the container with id af5ef8732db464515ef4129bed83c2202c334123d808c56bc596c82010613148 Oct 30 21:58:34.739071 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:34.739038 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-554f46-trdg6" event={"ID":"a42c755a-8283-4f86-86f4-9bdbe1753bdd","Type":"ContainerStarted","Data":"af5ef8732db464515ef4129bed83c2202c334123d808c56bc596c82010613148"} Oct 30 21:58:35.746589 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:35.746551 4560 generic.go:356] "Generic (PLEG): container finished" podID="a42c755a-8283-4f86-86f4-9bdbe1753bdd" containerID="c03d1e618d74c9b8ba2bcbf41721d61bab6e16df6402e21677ba65d80d2c025b" exitCode=0 Oct 30 21:58:35.746939 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:35.746599 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-554f46-trdg6" event={"ID":"a42c755a-8283-4f86-86f4-9bdbe1753bdd","Type":"ContainerDied","Data":"c03d1e618d74c9b8ba2bcbf41721d61bab6e16df6402e21677ba65d80d2c025b"} Oct 30 21:58:36.898225 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:36.898200 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-554f46-trdg6" Oct 30 21:58:36.966385 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:36.966333 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a42c755a-8283-4f86-86f4-9bdbe1753bdd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a42c755a-8283-4f86-86f4-9bdbe1753bdd-maas-api-openshift-ca\") pod \"a42c755a-8283-4f86-86f4-9bdbe1753bdd\" (UID: \"a42c755a-8283-4f86-86f4-9bdbe1753bdd\") " Oct 30 21:58:36.966563 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:36.966548 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a42c755a-8283-4f86-86f4-9bdbe1753bdd-kube-api-access-2l86w\" (UniqueName: \"kubernetes.io/projected/a42c755a-8283-4f86-86f4-9bdbe1753bdd-kube-api-access-2l86w\") pod \"a42c755a-8283-4f86-86f4-9bdbe1753bdd\" (UID: \"a42c755a-8283-4f86-86f4-9bdbe1753bdd\") " Oct 30 21:58:36.966617 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:36.966602 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a42c755a-8283-4f86-86f4-9bdbe1753bdd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a42c755a-8283-4f86-86f4-9bdbe1753bdd-maas-api-xks-ca\") pod \"a42c755a-8283-4f86-86f4-9bdbe1753bdd\" (UID: \"a42c755a-8283-4f86-86f4-9bdbe1753bdd\") " Oct 30 21:58:36.966699 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:36.966675 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a42c755a-8283-4f86-86f4-9bdbe1753bdd-maas-api-openshift-ca" pod "a42c755a-8283-4f86-86f4-9bdbe1753bdd" (UID: "a42c755a-8283-4f86-86f4-9bdbe1753bdd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:36.967055 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:36.967034 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a42c755a-8283-4f86-86f4-9bdbe1753bdd-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:36.969156 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:36.969131 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a42c755a-8283-4f86-86f4-9bdbe1753bdd-maas-api-xks-ca" pod "a42c755a-8283-4f86-86f4-9bdbe1753bdd" (UID: "a42c755a-8283-4f86-86f4-9bdbe1753bdd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:36.971586 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:36.971563 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a42c755a-8283-4f86-86f4-9bdbe1753bdd-kube-api-access-2l86w" pod "a42c755a-8283-4f86-86f4-9bdbe1753bdd" (UID: "a42c755a-8283-4f86-86f4-9bdbe1753bdd"). InnerVolumeSpecName "kube-api-access-2l86w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:37.069701 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:37.069633 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l86w\" (UniqueName: \"kubernetes.io/projected/a42c755a-8283-4f86-86f4-9bdbe1753bdd-kube-api-access-2l86w\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:37.069701 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:37.069662 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a42c755a-8283-4f86-86f4-9bdbe1753bdd-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:37.762849 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:37.762808 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-554f46-trdg6" event={"ID":"a42c755a-8283-4f86-86f4-9bdbe1753bdd","Type":"ContainerDied","Data":"af5ef8732db464515ef4129bed83c2202c334123d808c56bc596c82010613148"} Oct 30 21:58:37.763020 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:37.762870 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af5ef8732db464515ef4129bed83c2202c334123d808c56bc596c82010613148" Oct 30 21:58:37.763020 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:37.762827 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-554f46-trdg6" Oct 30 21:58:37.805231 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:37.805196 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h"] Oct 30 21:58:37.805438 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:37.805403 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" podUID="7a8c5ad5-8b50-4795-9596-d951de399015" containerName="istio-proxy" containerID="cri-o://fe5f65e4f226f1b28a6f9e2b05ee99a5d20b3c8f64cef8c788b2aa513711e589" gracePeriod=30 Oct 30 21:58:38.376800 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:38.376762 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf"] Oct 30 21:58:38.401093 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:38.401061 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk"] Oct 30 21:58:38.747716 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:38.747686 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-167561-dqlmp"] Oct 30 21:58:38.748168 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:38.748154 4560 cpu_manager.go:414] "containerMap: removing container" podUID="a42c755a-8283-4f86-86f4-9bdbe1753bdd" containerName="revoke-keys" Oct 30 21:58:38.748212 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:38.748169 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a42c755a-8283-4f86-86f4-9bdbe1753bdd" containerName="revoke-keys" Oct 30 21:58:38.748284 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:38.748273 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="a42c755a-8283-4f86-86f4-9bdbe1753bdd" containerName="revoke-keys" Oct 30 21:58:38.751734 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:38.751716 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-167561-dqlmp" Oct 30 21:58:38.802678 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:38.802644 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b831442-fe0b-42ea-8ea2-16313d31dd46-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-167561-dqlmp\" (UID: \"3b831442-fe0b-42ea-8ea2-16313d31dd46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-167561-dqlmp" Oct 30 21:58:38.802815 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:38.802693 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b831442-fe0b-42ea-8ea2-16313d31dd46-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-167561-dqlmp\" (UID: \"3b831442-fe0b-42ea-8ea2-16313d31dd46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-167561-dqlmp" Oct 30 21:58:38.802861 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:38.802815 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db2pm\" (UniqueName: \"kubernetes.io/projected/3b831442-fe0b-42ea-8ea2-16313d31dd46-kube-api-access-db2pm\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-167561-dqlmp\" (UID: \"3b831442-fe0b-42ea-8ea2-16313d31dd46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-167561-dqlmp" Oct 30 21:58:38.806023 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:38.805986 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:58:38.806178 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:38.806160 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:58:38.815136 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:38.815105 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-167561-dqlmp"] Oct 30 21:58:38.834539 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:38.834513 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:58:38.834660 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:38.834628 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:58:38.904170 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:38.904133 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b831442-fe0b-42ea-8ea2-16313d31dd46-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-167561-dqlmp\" (UID: \"3b831442-fe0b-42ea-8ea2-16313d31dd46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-167561-dqlmp" Oct 30 21:58:38.904320 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:38.904183 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b831442-fe0b-42ea-8ea2-16313d31dd46-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-167561-dqlmp\" (UID: \"3b831442-fe0b-42ea-8ea2-16313d31dd46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-167561-dqlmp" Oct 30 21:58:38.904320 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:38.904229 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db2pm\" (UniqueName: \"kubernetes.io/projected/3b831442-fe0b-42ea-8ea2-16313d31dd46-kube-api-access-db2pm\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-167561-dqlmp\" (UID: \"3b831442-fe0b-42ea-8ea2-16313d31dd46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-167561-dqlmp" Oct 30 21:58:38.904861 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:38.904842 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b831442-fe0b-42ea-8ea2-16313d31dd46-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-167561-dqlmp\" (UID: \"3b831442-fe0b-42ea-8ea2-16313d31dd46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-167561-dqlmp" Oct 30 21:58:38.906485 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:38.906461 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b831442-fe0b-42ea-8ea2-16313d31dd46-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-167561-dqlmp\" (UID: \"3b831442-fe0b-42ea-8ea2-16313d31dd46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-167561-dqlmp" Oct 30 21:58:38.963564 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:38.963533 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db2pm\" (UniqueName: \"kubernetes.io/projected/3b831442-fe0b-42ea-8ea2-16313d31dd46-kube-api-access-db2pm\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-167561-dqlmp\" (UID: \"3b831442-fe0b-42ea-8ea2-16313d31dd46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-167561-dqlmp" Oct 30 21:58:39.065130 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:39.065057 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-167561-dqlmp" Oct 30 21:58:39.438886 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:39.438814 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-167561-dqlmp"] Oct 30 21:58:39.443945 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:58:39.443919 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b831442_fe0b_42ea_8ea2_16313d31dd46.slice/crio-f110bad866257ac0b7dd692aac549758e1dfcc015f10ed73a4ab327c2052e215 WatchSource:0}: Error finding container f110bad866257ac0b7dd692aac549758e1dfcc015f10ed73a4ab327c2052e215: Status 404 returned error can't find the container with id f110bad866257ac0b7dd692aac549758e1dfcc015f10ed73a4ab327c2052e215 Oct 30 21:58:39.777687 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:39.777657 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-167561-dqlmp" event={"ID":"3b831442-fe0b-42ea-8ea2-16313d31dd46","Type":"ContainerStarted","Data":"711e8abb5e1d8ae1e5138ab28e1490859b8e300fea805e05659185181c557c7a"} Oct 30 21:58:39.777811 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:39.777714 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-167561-dqlmp" event={"ID":"3b831442-fe0b-42ea-8ea2-16313d31dd46","Type":"ContainerStarted","Data":"f110bad866257ac0b7dd692aac549758e1dfcc015f10ed73a4ab327c2052e215"} Oct 30 21:58:39.891726 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:39.891675 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-167561-dqlmp" podStartSLOduration=1.891662263 podStartE2EDuration="1.891662263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:39.891618798 +0000 UTC m=+2695.446318574" watchObservedRunningTime="2026-10-30 21:58:39.891662263 +0000 UTC m=+2695.446362038" Oct 30 21:58:40.666915 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:40.666842 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-27114d-chj7h"] Oct 30 21:58:40.670713 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:40.670693 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-27114d-chj7h" Oct 30 21:58:40.724427 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:40.724396 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9bc5ce-d25b-4010-ac88-48f27a08338c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-27114d-chj7h\" (UID: \"aa9bc5ce-d25b-4010-ac88-48f27a08338c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-27114d-chj7h" Oct 30 21:58:40.724566 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:40.724518 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9bc5ce-d25b-4010-ac88-48f27a08338c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-27114d-chj7h\" (UID: \"aa9bc5ce-d25b-4010-ac88-48f27a08338c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-27114d-chj7h" Oct 30 21:58:40.724801 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:40.724773 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv69v\" (UniqueName: \"kubernetes.io/projected/aa9bc5ce-d25b-4010-ac88-48f27a08338c-kube-api-access-mv69v\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-27114d-chj7h\" (UID: \"aa9bc5ce-d25b-4010-ac88-48f27a08338c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-27114d-chj7h" Oct 30 21:58:40.728314 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:40.728294 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-27114d-chj7h"] Oct 30 21:58:40.785229 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:40.785197 4560 generic.go:356] "Generic (PLEG): container finished" podID="3b831442-fe0b-42ea-8ea2-16313d31dd46" containerID="711e8abb5e1d8ae1e5138ab28e1490859b8e300fea805e05659185181c557c7a" exitCode=0 Oct 30 21:58:40.785405 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:40.785238 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-167561-dqlmp" event={"ID":"3b831442-fe0b-42ea-8ea2-16313d31dd46","Type":"ContainerDied","Data":"711e8abb5e1d8ae1e5138ab28e1490859b8e300fea805e05659185181c557c7a"} Oct 30 21:58:40.826381 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:40.826326 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9bc5ce-d25b-4010-ac88-48f27a08338c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-27114d-chj7h\" (UID: \"aa9bc5ce-d25b-4010-ac88-48f27a08338c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-27114d-chj7h" Oct 30 21:58:40.826497 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:40.826426 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9bc5ce-d25b-4010-ac88-48f27a08338c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-27114d-chj7h\" (UID: \"aa9bc5ce-d25b-4010-ac88-48f27a08338c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-27114d-chj7h" Oct 30 21:58:40.826589 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:40.826572 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv69v\" (UniqueName: \"kubernetes.io/projected/aa9bc5ce-d25b-4010-ac88-48f27a08338c-kube-api-access-mv69v\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-27114d-chj7h\" (UID: \"aa9bc5ce-d25b-4010-ac88-48f27a08338c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-27114d-chj7h" Oct 30 21:58:40.826994 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:40.826974 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9bc5ce-d25b-4010-ac88-48f27a08338c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-27114d-chj7h\" (UID: \"aa9bc5ce-d25b-4010-ac88-48f27a08338c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-27114d-chj7h" Oct 30 21:58:40.828757 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:40.828738 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9bc5ce-d25b-4010-ac88-48f27a08338c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-27114d-chj7h\" (UID: \"aa9bc5ce-d25b-4010-ac88-48f27a08338c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-27114d-chj7h" Oct 30 21:58:40.885683 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:40.885655 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv69v\" (UniqueName: \"kubernetes.io/projected/aa9bc5ce-d25b-4010-ac88-48f27a08338c-kube-api-access-mv69v\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-27114d-chj7h\" (UID: \"aa9bc5ce-d25b-4010-ac88-48f27a08338c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-27114d-chj7h" Oct 30 21:58:40.983797 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:40.983769 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-27114d-chj7h" Oct 30 21:58:41.366600 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:41.366526 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-27114d-chj7h"] Oct 30 21:58:41.371001 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:58:41.370972 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa9bc5ce_d25b_4010_ac88_48f27a08338c.slice/crio-d91f16e6430388d60b5c757f15169169cf0e30c500fe2d48c7a533f67b538d44 WatchSource:0}: Error finding container d91f16e6430388d60b5c757f15169169cf0e30c500fe2d48c7a533f67b538d44: Status 404 returned error can't find the container with id d91f16e6430388d60b5c757f15169169cf0e30c500fe2d48c7a533f67b538d44 Oct 30 21:58:41.792659 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:41.792628 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-27114d-chj7h" event={"ID":"aa9bc5ce-d25b-4010-ac88-48f27a08338c","Type":"ContainerStarted","Data":"3b99dcb2363e26e290ff2eb60c251d7e9893f5a9d3719dbe840c2c44bbe716fb"} Oct 30 21:58:41.793064 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:41.792678 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-27114d-chj7h" event={"ID":"aa9bc5ce-d25b-4010-ac88-48f27a08338c","Type":"ContainerStarted","Data":"d91f16e6430388d60b5c757f15169169cf0e30c500fe2d48c7a533f67b538d44"} Oct 30 21:58:41.922279 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:41.922230 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-27114d-chj7h" podStartSLOduration=1.922213511 podStartE2EDuration="1.922213511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:41.920244166 +0000 UTC m=+2697.474943942" watchObservedRunningTime="2026-10-30 21:58:41.922213511 +0000 UTC m=+2697.476913286" Oct 30 21:58:41.934832 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:41.934810 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-167561-dqlmp" Oct 30 21:58:42.040884 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:42.040856 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b831442-fe0b-42ea-8ea2-16313d31dd46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b831442-fe0b-42ea-8ea2-16313d31dd46-maas-api-xks-ca\") pod \"3b831442-fe0b-42ea-8ea2-16313d31dd46\" (UID: \"3b831442-fe0b-42ea-8ea2-16313d31dd46\") " Oct 30 21:58:42.041036 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:42.040978 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b831442-fe0b-42ea-8ea2-16313d31dd46-kube-api-access-db2pm\" (UniqueName: \"kubernetes.io/projected/3b831442-fe0b-42ea-8ea2-16313d31dd46-kube-api-access-db2pm\") pod \"3b831442-fe0b-42ea-8ea2-16313d31dd46\" (UID: \"3b831442-fe0b-42ea-8ea2-16313d31dd46\") " Oct 30 21:58:42.041097 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:42.041037 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b831442-fe0b-42ea-8ea2-16313d31dd46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b831442-fe0b-42ea-8ea2-16313d31dd46-maas-api-openshift-ca\") pod \"3b831442-fe0b-42ea-8ea2-16313d31dd46\" (UID: \"3b831442-fe0b-42ea-8ea2-16313d31dd46\") " Oct 30 21:58:42.041401 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:42.041375 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b831442-fe0b-42ea-8ea2-16313d31dd46-maas-api-openshift-ca" pod "3b831442-fe0b-42ea-8ea2-16313d31dd46" (UID: "3b831442-fe0b-42ea-8ea2-16313d31dd46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:42.042921 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:42.042896 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b831442-fe0b-42ea-8ea2-16313d31dd46-maas-api-xks-ca" pod "3b831442-fe0b-42ea-8ea2-16313d31dd46" (UID: "3b831442-fe0b-42ea-8ea2-16313d31dd46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:42.043024 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:42.043007 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b831442-fe0b-42ea-8ea2-16313d31dd46-kube-api-access-db2pm" pod "3b831442-fe0b-42ea-8ea2-16313d31dd46" (UID: "3b831442-fe0b-42ea-8ea2-16313d31dd46"). InnerVolumeSpecName "kube-api-access-db2pm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:42.142876 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:42.142848 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db2pm\" (UniqueName: \"kubernetes.io/projected/3b831442-fe0b-42ea-8ea2-16313d31dd46-kube-api-access-db2pm\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:42.142876 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:42.142874 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b831442-fe0b-42ea-8ea2-16313d31dd46-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:42.143066 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:42.142884 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b831442-fe0b-42ea-8ea2-16313d31dd46-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:42.800277 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:42.800250 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-167561-dqlmp" Oct 30 21:58:42.800737 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:42.800288 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-167561-dqlmp" event={"ID":"3b831442-fe0b-42ea-8ea2-16313d31dd46","Type":"ContainerDied","Data":"f110bad866257ac0b7dd692aac549758e1dfcc015f10ed73a4ab327c2052e215"} Oct 30 21:58:42.800737 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:42.800335 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f110bad866257ac0b7dd692aac549758e1dfcc015f10ed73a4ab327c2052e215" Oct 30 21:58:42.801846 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:42.801820 4560 generic.go:356] "Generic (PLEG): container finished" podID="aa9bc5ce-d25b-4010-ac88-48f27a08338c" containerID="3b99dcb2363e26e290ff2eb60c251d7e9893f5a9d3719dbe840c2c44bbe716fb" exitCode=0 Oct 30 21:58:42.801965 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:42.801900 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-27114d-chj7h" event={"ID":"aa9bc5ce-d25b-4010-ac88-48f27a08338c","Type":"ContainerDied","Data":"3b99dcb2363e26e290ff2eb60c251d7e9893f5a9d3719dbe840c2c44bbe716fb"} Oct 30 21:58:43.079051 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:43.079027 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" Oct 30 21:58:43.154147 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:43.154120 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a8c5ad5-8b50-4795-9596-d951de399015-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a8c5ad5-8b50-4795-9596-d951de399015-workload-certs\") pod \"7a8c5ad5-8b50-4795-9596-d951de399015\" (UID: \"7a8c5ad5-8b50-4795-9596-d951de399015\") " Oct 30 21:58:43.154322 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:43.154172 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a8c5ad5-8b50-4795-9596-d951de399015-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a8c5ad5-8b50-4795-9596-d951de399015-istio-podinfo\") pod \"7a8c5ad5-8b50-4795-9596-d951de399015\" (UID: \"7a8c5ad5-8b50-4795-9596-d951de399015\") " Oct 30 21:58:43.154322 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:43.154201 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a8c5ad5-8b50-4795-9596-d951de399015-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a8c5ad5-8b50-4795-9596-d951de399015-istiod-ca-cert\") pod \"7a8c5ad5-8b50-4795-9596-d951de399015\" (UID: \"7a8c5ad5-8b50-4795-9596-d951de399015\") " Oct 30 21:58:43.154322 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:43.154257 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a8c5ad5-8b50-4795-9596-d951de399015-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8c5ad5-8b50-4795-9596-d951de399015-workload-socket\") pod \"7a8c5ad5-8b50-4795-9596-d951de399015\" (UID: \"7a8c5ad5-8b50-4795-9596-d951de399015\") " Oct 30 21:58:43.154515 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:43.154382 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a8c5ad5-8b50-4795-9596-d951de399015-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a8c5ad5-8b50-4795-9596-d951de399015-istio-envoy\") pod \"7a8c5ad5-8b50-4795-9596-d951de399015\" (UID: \"7a8c5ad5-8b50-4795-9596-d951de399015\") " Oct 30 21:58:43.154515 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:43.154468 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a8c5ad5-8b50-4795-9596-d951de399015-workload-certs" pod "7a8c5ad5-8b50-4795-9596-d951de399015" (UID: "7a8c5ad5-8b50-4795-9596-d951de399015"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:43.154515 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:43.154487 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a8c5ad5-8b50-4795-9596-d951de399015-istio-token\" (UniqueName: \"kubernetes.io/projected/7a8c5ad5-8b50-4795-9596-d951de399015-istio-token\") pod \"7a8c5ad5-8b50-4795-9596-d951de399015\" (UID: \"7a8c5ad5-8b50-4795-9596-d951de399015\") " Oct 30 21:58:43.154659 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:43.154528 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a8c5ad5-8b50-4795-9596-d951de399015-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8c5ad5-8b50-4795-9596-d951de399015-credential-socket\") pod \"7a8c5ad5-8b50-4795-9596-d951de399015\" (UID: \"7a8c5ad5-8b50-4795-9596-d951de399015\") " Oct 30 21:58:43.154659 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:43.154559 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a8c5ad5-8b50-4795-9596-d951de399015-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a8c5ad5-8b50-4795-9596-d951de399015-istio-data\") pod \"7a8c5ad5-8b50-4795-9596-d951de399015\" (UID: \"7a8c5ad5-8b50-4795-9596-d951de399015\") " Oct 30 21:58:43.154659 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:43.154599 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a8c5ad5-8b50-4795-9596-d951de399015-workload-socket" pod "7a8c5ad5-8b50-4795-9596-d951de399015" (UID: "7a8c5ad5-8b50-4795-9596-d951de399015"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:43.154820 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:43.154674 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a8c5ad5-8b50-4795-9596-d951de399015-istiod-ca-cert" pod "7a8c5ad5-8b50-4795-9596-d951de399015" (UID: "7a8c5ad5-8b50-4795-9596-d951de399015"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:43.154820 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:43.154689 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a8c5ad5-8b50-4795-9596-d951de399015-kube-api-access-wcwd7\" (UniqueName: \"kubernetes.io/projected/7a8c5ad5-8b50-4795-9596-d951de399015-kube-api-access-wcwd7\") pod \"7a8c5ad5-8b50-4795-9596-d951de399015\" (UID: \"7a8c5ad5-8b50-4795-9596-d951de399015\") " Oct 30 21:58:43.154820 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:43.154800 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a8c5ad5-8b50-4795-9596-d951de399015-credential-socket" pod "7a8c5ad5-8b50-4795-9596-d951de399015" (UID: "7a8c5ad5-8b50-4795-9596-d951de399015"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:43.154973 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:43.154877 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a8c5ad5-8b50-4795-9596-d951de399015-istio-data" pod "7a8c5ad5-8b50-4795-9596-d951de399015" (UID: "7a8c5ad5-8b50-4795-9596-d951de399015"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:43.155188 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:43.155169 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8c5ad5-8b50-4795-9596-d951de399015-workload-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:43.155270 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:43.155192 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8c5ad5-8b50-4795-9596-d951de399015-credential-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:43.155270 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:43.155205 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a8c5ad5-8b50-4795-9596-d951de399015-istio-data\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:43.155270 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:43.155218 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a8c5ad5-8b50-4795-9596-d951de399015-workload-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:43.155270 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:43.155230 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a8c5ad5-8b50-4795-9596-d951de399015-istiod-ca-cert\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:43.156559 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:43.156539 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a8c5ad5-8b50-4795-9596-d951de399015-istio-podinfo" pod "7a8c5ad5-8b50-4795-9596-d951de399015" (UID: "7a8c5ad5-8b50-4795-9596-d951de399015"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:58:43.156802 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:43.156784 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a8c5ad5-8b50-4795-9596-d951de399015-kube-api-access-wcwd7" pod "7a8c5ad5-8b50-4795-9596-d951de399015" (UID: "7a8c5ad5-8b50-4795-9596-d951de399015"). InnerVolumeSpecName "kube-api-access-wcwd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:43.156949 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:43.156929 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a8c5ad5-8b50-4795-9596-d951de399015-istio-envoy" pod "7a8c5ad5-8b50-4795-9596-d951de399015" (UID: "7a8c5ad5-8b50-4795-9596-d951de399015"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:43.157010 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:43.156932 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a8c5ad5-8b50-4795-9596-d951de399015-istio-token" pod "7a8c5ad5-8b50-4795-9596-d951de399015" (UID: "7a8c5ad5-8b50-4795-9596-d951de399015"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:43.256423 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:43.256395 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcwd7\" (UniqueName: \"kubernetes.io/projected/7a8c5ad5-8b50-4795-9596-d951de399015-kube-api-access-wcwd7\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:43.256423 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:43.256420 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a8c5ad5-8b50-4795-9596-d951de399015-istio-podinfo\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:43.256601 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:43.256430 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a8c5ad5-8b50-4795-9596-d951de399015-istio-envoy\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:43.256601 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:43.256438 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a8c5ad5-8b50-4795-9596-d951de399015-istio-token\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:43.812399 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:43.812361 4560 generic.go:356] "Generic (PLEG): container finished" podID="7a8c5ad5-8b50-4795-9596-d951de399015" containerID="fe5f65e4f226f1b28a6f9e2b05ee99a5d20b3c8f64cef8c788b2aa513711e589" exitCode=0 Oct 30 21:58:43.812845 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:43.812406 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" Oct 30 21:58:43.812845 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:43.812454 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" event={"ID":"7a8c5ad5-8b50-4795-9596-d951de399015","Type":"ContainerDied","Data":"fe5f65e4f226f1b28a6f9e2b05ee99a5d20b3c8f64cef8c788b2aa513711e589"} Oct 30 21:58:43.812845 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:43.812508 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h" event={"ID":"7a8c5ad5-8b50-4795-9596-d951de399015","Type":"ContainerDied","Data":"00a95dbdf1102f62e85004d83980c174685bdcd82f64abee736e575656d7304b"} Oct 30 21:58:43.812845 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:43.812536 4560 scope.go:122] "RemoveContainer" containerID="fe5f65e4f226f1b28a6f9e2b05ee99a5d20b3c8f64cef8c788b2aa513711e589" Oct 30 21:58:43.822899 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:43.822878 4560 scope.go:122] "RemoveContainer" containerID="ddf18f19c908cd1807a1db3b7a9a0ff05d35e44448bf0556ee0ea9d41d3c2276" Oct 30 21:58:43.831578 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:43.831550 4560 scope.go:122] "RemoveContainer" containerID="fe5f65e4f226f1b28a6f9e2b05ee99a5d20b3c8f64cef8c788b2aa513711e589" Oct 30 21:58:43.831898 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:58:43.831878 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe5f65e4f226f1b28a6f9e2b05ee99a5d20b3c8f64cef8c788b2aa513711e589\": container with ID starting with fe5f65e4f226f1b28a6f9e2b05ee99a5d20b3c8f64cef8c788b2aa513711e589 not found: ID does not exist" containerID="fe5f65e4f226f1b28a6f9e2b05ee99a5d20b3c8f64cef8c788b2aa513711e589" Oct 30 21:58:43.831972 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:43.831909 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe5f65e4f226f1b28a6f9e2b05ee99a5d20b3c8f64cef8c788b2aa513711e589"} err="failed to get container status \"fe5f65e4f226f1b28a6f9e2b05ee99a5d20b3c8f64cef8c788b2aa513711e589\": rpc error: code = NotFound desc = could not find container \"fe5f65e4f226f1b28a6f9e2b05ee99a5d20b3c8f64cef8c788b2aa513711e589\": container with ID starting with fe5f65e4f226f1b28a6f9e2b05ee99a5d20b3c8f64cef8c788b2aa513711e589 not found: ID does not exist" Oct 30 21:58:43.831972 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:43.831932 4560 scope.go:122] "RemoveContainer" containerID="ddf18f19c908cd1807a1db3b7a9a0ff05d35e44448bf0556ee0ea9d41d3c2276" Oct 30 21:58:43.832209 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:58:43.832192 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddf18f19c908cd1807a1db3b7a9a0ff05d35e44448bf0556ee0ea9d41d3c2276\": container with ID starting with ddf18f19c908cd1807a1db3b7a9a0ff05d35e44448bf0556ee0ea9d41d3c2276 not found: ID does not exist" containerID="ddf18f19c908cd1807a1db3b7a9a0ff05d35e44448bf0556ee0ea9d41d3c2276" Oct 30 21:58:43.832272 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:43.832220 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddf18f19c908cd1807a1db3b7a9a0ff05d35e44448bf0556ee0ea9d41d3c2276"} err="failed to get container status \"ddf18f19c908cd1807a1db3b7a9a0ff05d35e44448bf0556ee0ea9d41d3c2276\": rpc error: code = NotFound desc = could not find container \"ddf18f19c908cd1807a1db3b7a9a0ff05d35e44448bf0556ee0ea9d41d3c2276\": container with ID starting with ddf18f19c908cd1807a1db3b7a9a0ff05d35e44448bf0556ee0ea9d41d3c2276 not found: ID does not exist" Oct 30 21:58:43.963791 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:43.963766 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-27114d-chj7h" Oct 30 21:58:44.000308 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:44.000278 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h"] Oct 30 21:58:44.046771 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:44.046744 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2aa268-openshift-default-76644cff95-79v4h"] Oct 30 21:58:44.064754 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:44.064695 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa9bc5ce-d25b-4010-ac88-48f27a08338c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9bc5ce-d25b-4010-ac88-48f27a08338c-maas-api-xks-ca\") pod \"aa9bc5ce-d25b-4010-ac88-48f27a08338c\" (UID: \"aa9bc5ce-d25b-4010-ac88-48f27a08338c\") " Oct 30 21:58:44.064892 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:44.064876 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa9bc5ce-d25b-4010-ac88-48f27a08338c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9bc5ce-d25b-4010-ac88-48f27a08338c-maas-api-openshift-ca\") pod \"aa9bc5ce-d25b-4010-ac88-48f27a08338c\" (UID: \"aa9bc5ce-d25b-4010-ac88-48f27a08338c\") " Oct 30 21:58:44.064961 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:44.064915 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa9bc5ce-d25b-4010-ac88-48f27a08338c-kube-api-access-mv69v\" (UniqueName: \"kubernetes.io/projected/aa9bc5ce-d25b-4010-ac88-48f27a08338c-kube-api-access-mv69v\") pod \"aa9bc5ce-d25b-4010-ac88-48f27a08338c\" (UID: \"aa9bc5ce-d25b-4010-ac88-48f27a08338c\") " Oct 30 21:58:44.065196 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:44.065176 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa9bc5ce-d25b-4010-ac88-48f27a08338c-maas-api-openshift-ca" pod "aa9bc5ce-d25b-4010-ac88-48f27a08338c" (UID: "aa9bc5ce-d25b-4010-ac88-48f27a08338c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:44.065315 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:44.065303 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9bc5ce-d25b-4010-ac88-48f27a08338c-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:44.066646 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:44.066628 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa9bc5ce-d25b-4010-ac88-48f27a08338c-maas-api-xks-ca" pod "aa9bc5ce-d25b-4010-ac88-48f27a08338c" (UID: "aa9bc5ce-d25b-4010-ac88-48f27a08338c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:44.066830 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:44.066812 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa9bc5ce-d25b-4010-ac88-48f27a08338c-kube-api-access-mv69v" pod "aa9bc5ce-d25b-4010-ac88-48f27a08338c" (UID: "aa9bc5ce-d25b-4010-ac88-48f27a08338c"). InnerVolumeSpecName "kube-api-access-mv69v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:44.166372 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:44.166330 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv69v\" (UniqueName: \"kubernetes.io/projected/aa9bc5ce-d25b-4010-ac88-48f27a08338c-kube-api-access-mv69v\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:44.166372 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:44.166369 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9bc5ce-d25b-4010-ac88-48f27a08338c-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:44.577298 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:44.577270 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a8c5ad5-8b50-4795-9596-d951de399015" path="/var/lib/kubelet/pods/7a8c5ad5-8b50-4795-9596-d951de399015/volumes" Oct 30 21:58:44.820536 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:44.820501 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-27114d-chj7h" Oct 30 21:58:44.820536 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:44.820527 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-27114d-chj7h" event={"ID":"aa9bc5ce-d25b-4010-ac88-48f27a08338c","Type":"ContainerDied","Data":"d91f16e6430388d60b5c757f15169169cf0e30c500fe2d48c7a533f67b538d44"} Oct 30 21:58:44.821112 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:44.820579 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d91f16e6430388d60b5c757f15169169cf0e30c500fe2d48c7a533f67b538d44" Oct 30 21:58:47.158915 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:47.158876 4560 scope.go:122] "RemoveContainer" containerID="46bca71a1bfc89a84503754a38f887d8963047d714ff552a2c5047b2e8908305" Oct 30 21:58:49.732169 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:58:49.732135 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:58:53.398374 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.398319 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" podUID="0155a02e-8df7-4847-b40f-b8d9e4bac8c6" containerName="main" containerID="cri-o://7cb8434476e1f9021cba0ba7392104f1bf5032786bce784254ed2755dfc5c9f4" gracePeriod=45 Oct 30 21:58:53.419909 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.419876 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" podUID="0ff8ae26-4364-4c4e-9285-721d1a486a9b" containerName="main" containerID="cri-o://c1e96ba62d7d61acfa14242db5c74d9ccef1d0cab5ff66e0f1b244c548150fde" gracePeriod=45 Oct 30 21:58:53.679365 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.679316 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" Oct 30 21:58:53.709991 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.709966 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" Oct 30 21:58:53.763494 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.763465 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-model-cache\") pod \"0155a02e-8df7-4847-b40f-b8d9e4bac8c6\" (UID: \"0155a02e-8df7-4847-b40f-b8d9e4bac8c6\") " Oct 30 21:58:53.763637 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.763528 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-home\" (UniqueName: \"kubernetes.io/empty-dir/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-home\") pod \"0155a02e-8df7-4847-b40f-b8d9e4bac8c6\" (UID: \"0155a02e-8df7-4847-b40f-b8d9e4bac8c6\") " Oct 30 21:58:53.763637 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.763601 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff8ae26-4364-4c4e-9285-721d1a486a9b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ff8ae26-4364-4c4e-9285-721d1a486a9b-tmp-dir\") pod \"0ff8ae26-4364-4c4e-9285-721d1a486a9b\" (UID: \"0ff8ae26-4364-4c4e-9285-721d1a486a9b\") " Oct 30 21:58:53.763637 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.763625 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-tmp-dir\") pod \"0155a02e-8df7-4847-b40f-b8d9e4bac8c6\" (UID: \"0155a02e-8df7-4847-b40f-b8d9e4bac8c6\") " Oct 30 21:58:53.763789 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.763680 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff8ae26-4364-4c4e-9285-721d1a486a9b-home\" (UniqueName: \"kubernetes.io/empty-dir/0ff8ae26-4364-4c4e-9285-721d1a486a9b-home\") pod \"0ff8ae26-4364-4c4e-9285-721d1a486a9b\" (UID: \"0ff8ae26-4364-4c4e-9285-721d1a486a9b\") " Oct 30 21:58:53.763789 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.763719 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-kube-api-access-qclqz\" (UniqueName: \"kubernetes.io/projected/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-kube-api-access-qclqz\") pod \"0155a02e-8df7-4847-b40f-b8d9e4bac8c6\" (UID: \"0155a02e-8df7-4847-b40f-b8d9e4bac8c6\") " Oct 30 21:58:53.763789 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.763710 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-model-cache" pod "0155a02e-8df7-4847-b40f-b8d9e4bac8c6" (UID: "0155a02e-8df7-4847-b40f-b8d9e4bac8c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:53.763789 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.763743 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-home" pod "0155a02e-8df7-4847-b40f-b8d9e4bac8c6" (UID: "0155a02e-8df7-4847-b40f-b8d9e4bac8c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:53.763789 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.763771 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ff8ae26-4364-4c4e-9285-721d1a486a9b-kube-api-access-qxhn8\" (UniqueName: \"kubernetes.io/projected/0ff8ae26-4364-4c4e-9285-721d1a486a9b-kube-api-access-qxhn8\") pod \"0ff8ae26-4364-4c4e-9285-721d1a486a9b\" (UID: \"0ff8ae26-4364-4c4e-9285-721d1a486a9b\") " Oct 30 21:58:53.764034 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.763824 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-tls-certs\" (UniqueName: \"kubernetes.io/secret/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-tls-certs\") pod \"0155a02e-8df7-4847-b40f-b8d9e4bac8c6\" (UID: \"0155a02e-8df7-4847-b40f-b8d9e4bac8c6\") " Oct 30 21:58:53.764034 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.763868 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ff8ae26-4364-4c4e-9285-721d1a486a9b-tls-certs\" (UniqueName: \"kubernetes.io/secret/0ff8ae26-4364-4c4e-9285-721d1a486a9b-tls-certs\") pod \"0ff8ae26-4364-4c4e-9285-721d1a486a9b\" (UID: \"0ff8ae26-4364-4c4e-9285-721d1a486a9b\") " Oct 30 21:58:53.764034 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.763865 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff8ae26-4364-4c4e-9285-721d1a486a9b-tmp-dir" pod "0ff8ae26-4364-4c4e-9285-721d1a486a9b" (UID: "0ff8ae26-4364-4c4e-9285-721d1a486a9b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:53.764034 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.763869 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-tmp-dir" pod "0155a02e-8df7-4847-b40f-b8d9e4bac8c6" (UID: "0155a02e-8df7-4847-b40f-b8d9e4bac8c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:53.764034 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.763899 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff8ae26-4364-4c4e-9285-721d1a486a9b-home" pod "0ff8ae26-4364-4c4e-9285-721d1a486a9b" (UID: "0ff8ae26-4364-4c4e-9285-721d1a486a9b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:53.764034 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.763906 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff8ae26-4364-4c4e-9285-721d1a486a9b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ff8ae26-4364-4c4e-9285-721d1a486a9b-model-cache\") pod \"0ff8ae26-4364-4c4e-9285-721d1a486a9b\" (UID: \"0ff8ae26-4364-4c4e-9285-721d1a486a9b\") " Oct 30 21:58:53.764034 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.763986 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-dshm\") pod \"0155a02e-8df7-4847-b40f-b8d9e4bac8c6\" (UID: \"0155a02e-8df7-4847-b40f-b8d9e4bac8c6\") " Oct 30 21:58:53.764554 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.764083 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff8ae26-4364-4c4e-9285-721d1a486a9b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ff8ae26-4364-4c4e-9285-721d1a486a9b-dshm\") pod \"0ff8ae26-4364-4c4e-9285-721d1a486a9b\" (UID: \"0ff8ae26-4364-4c4e-9285-721d1a486a9b\") " Oct 30 21:58:53.764554 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.764427 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff8ae26-4364-4c4e-9285-721d1a486a9b-model-cache" pod "0ff8ae26-4364-4c4e-9285-721d1a486a9b" (UID: "0ff8ae26-4364-4c4e-9285-721d1a486a9b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:53.764651 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.764638 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ff8ae26-4364-4c4e-9285-721d1a486a9b-home\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:53.764704 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.764658 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ff8ae26-4364-4c4e-9285-721d1a486a9b-model-cache\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:53.764704 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.764672 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-model-cache\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:53.764704 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.764687 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-home\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:53.764704 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.764700 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ff8ae26-4364-4c4e-9285-721d1a486a9b-tmp-dir\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:53.764909 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.764714 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-tmp-dir\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:53.766322 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.766295 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-kube-api-access-qclqz" pod "0155a02e-8df7-4847-b40f-b8d9e4bac8c6" (UID: "0155a02e-8df7-4847-b40f-b8d9e4bac8c6"). InnerVolumeSpecName "kube-api-access-qclqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:53.766447 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.766377 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-tls-certs" pod "0155a02e-8df7-4847-b40f-b8d9e4bac8c6" (UID: "0155a02e-8df7-4847-b40f-b8d9e4bac8c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:53.766594 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.766560 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ff8ae26-4364-4c4e-9285-721d1a486a9b-tls-certs" pod "0ff8ae26-4364-4c4e-9285-721d1a486a9b" (UID: "0ff8ae26-4364-4c4e-9285-721d1a486a9b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:53.766594 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.766574 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff8ae26-4364-4c4e-9285-721d1a486a9b-kube-api-access-qxhn8" pod "0ff8ae26-4364-4c4e-9285-721d1a486a9b" (UID: "0ff8ae26-4364-4c4e-9285-721d1a486a9b"). InnerVolumeSpecName "kube-api-access-qxhn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:53.766725 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.766643 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-dshm" pod "0155a02e-8df7-4847-b40f-b8d9e4bac8c6" (UID: "0155a02e-8df7-4847-b40f-b8d9e4bac8c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:53.766764 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.766754 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff8ae26-4364-4c4e-9285-721d1a486a9b-dshm" pod "0ff8ae26-4364-4c4e-9285-721d1a486a9b" (UID: "0ff8ae26-4364-4c4e-9285-721d1a486a9b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:53.866086 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.866050 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qclqz\" (UniqueName: \"kubernetes.io/projected/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-kube-api-access-qclqz\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:53.866086 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.866079 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxhn8\" (UniqueName: \"kubernetes.io/projected/0ff8ae26-4364-4c4e-9285-721d1a486a9b-kube-api-access-qxhn8\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:53.866314 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.866094 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-tls-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:53.866314 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.866104 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ff8ae26-4364-4c4e-9285-721d1a486a9b-tls-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:53.866314 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.866113 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0155a02e-8df7-4847-b40f-b8d9e4bac8c6-dshm\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:53.866314 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.866121 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ff8ae26-4364-4c4e-9285-721d1a486a9b-dshm\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:53.904532 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.904440 4560 generic.go:356] "Generic (PLEG): container finished" podID="0155a02e-8df7-4847-b40f-b8d9e4bac8c6" containerID="7cb8434476e1f9021cba0ba7392104f1bf5032786bce784254ed2755dfc5c9f4" exitCode=0 Oct 30 21:58:53.904532 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.904502 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" Oct 30 21:58:53.904717 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.904527 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" event={"ID":"0155a02e-8df7-4847-b40f-b8d9e4bac8c6","Type":"ContainerDied","Data":"7cb8434476e1f9021cba0ba7392104f1bf5032786bce784254ed2755dfc5c9f4"} Oct 30 21:58:53.904717 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.904597 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf" event={"ID":"0155a02e-8df7-4847-b40f-b8d9e4bac8c6","Type":"ContainerDied","Data":"4707bdff7c862e4b927c08e73236972d6cbbe7636003dbcecd8ecfb9750b0d26"} Oct 30 21:58:53.904717 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.904621 4560 scope.go:122] "RemoveContainer" containerID="7cb8434476e1f9021cba0ba7392104f1bf5032786bce784254ed2755dfc5c9f4" Oct 30 21:58:53.906258 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.906236 4560 generic.go:356] "Generic (PLEG): container finished" podID="0ff8ae26-4364-4c4e-9285-721d1a486a9b" containerID="c1e96ba62d7d61acfa14242db5c74d9ccef1d0cab5ff66e0f1b244c548150fde" exitCode=0 Oct 30 21:58:53.906406 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.906291 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" event={"ID":"0ff8ae26-4364-4c4e-9285-721d1a486a9b","Type":"ContainerDied","Data":"c1e96ba62d7d61acfa14242db5c74d9ccef1d0cab5ff66e0f1b244c548150fde"} Oct 30 21:58:53.906406 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.906318 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" event={"ID":"0ff8ae26-4364-4c4e-9285-721d1a486a9b","Type":"ContainerDied","Data":"fca0e6af1fd8fc23e2487862fdb155524671e72eb2dc639e2ca6a01dddae7ca2"} Oct 30 21:58:53.906406 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.906394 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk" Oct 30 21:58:53.915794 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.915775 4560 scope.go:122] "RemoveContainer" containerID="7cb8434476e1f9021cba0ba7392104f1bf5032786bce784254ed2755dfc5c9f4" Oct 30 21:58:53.916191 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:58:53.916166 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cb8434476e1f9021cba0ba7392104f1bf5032786bce784254ed2755dfc5c9f4\": container with ID starting with 7cb8434476e1f9021cba0ba7392104f1bf5032786bce784254ed2755dfc5c9f4 not found: ID does not exist" containerID="7cb8434476e1f9021cba0ba7392104f1bf5032786bce784254ed2755dfc5c9f4" Oct 30 21:58:53.916323 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.916200 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cb8434476e1f9021cba0ba7392104f1bf5032786bce784254ed2755dfc5c9f4"} err="failed to get container status \"7cb8434476e1f9021cba0ba7392104f1bf5032786bce784254ed2755dfc5c9f4\": rpc error: code = NotFound desc = could not find container \"7cb8434476e1f9021cba0ba7392104f1bf5032786bce784254ed2755dfc5c9f4\": container with ID starting with 7cb8434476e1f9021cba0ba7392104f1bf5032786bce784254ed2755dfc5c9f4 not found: ID does not exist" Oct 30 21:58:53.916323 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.916224 4560 scope.go:122] "RemoveContainer" containerID="c1e96ba62d7d61acfa14242db5c74d9ccef1d0cab5ff66e0f1b244c548150fde" Oct 30 21:58:53.927123 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.926927 4560 scope.go:122] "RemoveContainer" containerID="c1e96ba62d7d61acfa14242db5c74d9ccef1d0cab5ff66e0f1b244c548150fde" Oct 30 21:58:53.927856 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:58:53.927834 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1e96ba62d7d61acfa14242db5c74d9ccef1d0cab5ff66e0f1b244c548150fde\": container with ID starting with c1e96ba62d7d61acfa14242db5c74d9ccef1d0cab5ff66e0f1b244c548150fde not found: ID does not exist" containerID="c1e96ba62d7d61acfa14242db5c74d9ccef1d0cab5ff66e0f1b244c548150fde" Oct 30 21:58:53.927942 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:53.927867 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1e96ba62d7d61acfa14242db5c74d9ccef1d0cab5ff66e0f1b244c548150fde"} err="failed to get container status \"c1e96ba62d7d61acfa14242db5c74d9ccef1d0cab5ff66e0f1b244c548150fde\": rpc error: code = NotFound desc = could not find container \"c1e96ba62d7d61acfa14242db5c74d9ccef1d0cab5ff66e0f1b244c548150fde\": container with ID starting with c1e96ba62d7d61acfa14242db5c74d9ccef1d0cab5ff66e0f1b244c548150fde not found: ID does not exist" Oct 30 21:58:54.145769 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:54.145733 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk"] Oct 30 21:58:54.158165 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:54.158087 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-554f46/simulator-w4-554f46-kserve-c6f5f9bf7-l8zqk"] Oct 30 21:58:54.264824 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:54.264790 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf"] Oct 30 21:58:54.270826 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:54.270791 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-554f46/premium-w4-554f46-kserve-6848cd5dfb-8j9vf"] Oct 30 21:58:54.577483 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:54.577448 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0155a02e-8df7-4847-b40f-b8d9e4bac8c6" path="/var/lib/kubelet/pods/0155a02e-8df7-4847-b40f-b8d9e4bac8c6/volumes" Oct 30 21:58:54.577955 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:54.577934 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ff8ae26-4364-4c4e-9285-721d1a486a9b" path="/var/lib/kubelet/pods/0ff8ae26-4364-4c4e-9285-721d1a486a9b/volumes" Oct 30 21:58:59.893036 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:59.892998 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71c0c7-ptgs7"] Oct 30 21:58:59.893553 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:59.893450 4560 cpu_manager.go:414] "containerMap: removing container" podUID="7a8c5ad5-8b50-4795-9596-d951de399015" containerName="istio-proxy" Oct 30 21:58:59.893553 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:59.893463 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a8c5ad5-8b50-4795-9596-d951de399015" containerName="istio-proxy" Oct 30 21:58:59.893553 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:59.893481 4560 cpu_manager.go:414] "containerMap: removing container" podUID="0ff8ae26-4364-4c4e-9285-721d1a486a9b" containerName="main" Oct 30 21:58:59.893553 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:59.893486 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ff8ae26-4364-4c4e-9285-721d1a486a9b" containerName="main" Oct 30 21:58:59.893553 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:59.893508 4560 cpu_manager.go:414] "containerMap: removing container" podUID="aa9bc5ce-d25b-4010-ac88-48f27a08338c" containerName="revoke-keys" Oct 30 21:58:59.893553 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:59.893513 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa9bc5ce-d25b-4010-ac88-48f27a08338c" containerName="revoke-keys" Oct 30 21:58:59.893553 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:59.893524 4560 cpu_manager.go:414] "containerMap: removing container" podUID="0155a02e-8df7-4847-b40f-b8d9e4bac8c6" containerName="main" Oct 30 21:58:59.893553 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:59.893529 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0155a02e-8df7-4847-b40f-b8d9e4bac8c6" containerName="main" Oct 30 21:58:59.893553 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:59.893535 4560 cpu_manager.go:414] "containerMap: removing container" podUID="3b831442-fe0b-42ea-8ea2-16313d31dd46" containerName="revoke-keys" Oct 30 21:58:59.893553 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:59.893540 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b831442-fe0b-42ea-8ea2-16313d31dd46" containerName="revoke-keys" Oct 30 21:58:59.893907 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:59.893610 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa9bc5ce-d25b-4010-ac88-48f27a08338c" containerName="revoke-keys" Oct 30 21:58:59.893907 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:59.893620 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="0155a02e-8df7-4847-b40f-b8d9e4bac8c6" containerName="main" Oct 30 21:58:59.893907 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:59.893626 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b831442-fe0b-42ea-8ea2-16313d31dd46" containerName="revoke-keys" Oct 30 21:58:59.893907 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:59.893634 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a8c5ad5-8b50-4795-9596-d951de399015" containerName="istio-proxy" Oct 30 21:58:59.893907 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:59.893640 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a8c5ad5-8b50-4795-9596-d951de399015" containerName="istio-proxy" Oct 30 21:58:59.893907 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:59.893648 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ff8ae26-4364-4c4e-9285-721d1a486a9b" containerName="main" Oct 30 21:58:59.898477 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:59.898461 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71c0c7-ptgs7" Oct 30 21:58:59.952052 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:59.952017 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71c0c7-ptgs7"] Oct 30 21:58:59.958948 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:59.958920 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v6z9\" (UniqueName: \"kubernetes.io/projected/d7b824c4-575b-4c1a-bcf4-0f833dd21f70-kube-api-access-9v6z9\") pod \"maas-api-revoke-keys-e2e-api-a-71c0c7-ptgs7\" (UID: \"d7b824c4-575b-4c1a-bcf4-0f833dd21f70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71c0c7-ptgs7" Oct 30 21:58:59.959085 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:59.958967 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7b824c4-575b-4c1a-bcf4-0f833dd21f70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-71c0c7-ptgs7\" (UID: \"d7b824c4-575b-4c1a-bcf4-0f833dd21f70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71c0c7-ptgs7" Oct 30 21:58:59.959085 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:59.959039 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7b824c4-575b-4c1a-bcf4-0f833dd21f70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-71c0c7-ptgs7\" (UID: \"d7b824c4-575b-4c1a-bcf4-0f833dd21f70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71c0c7-ptgs7" Oct 30 21:58:59.995616 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:59.995588 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:58:59.995793 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:59.995777 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:58:59.995863 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:59.995790 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:58:59.995863 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:58:59.995851 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:59:00.060066 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:00.060030 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v6z9\" (UniqueName: \"kubernetes.io/projected/d7b824c4-575b-4c1a-bcf4-0f833dd21f70-kube-api-access-9v6z9\") pod \"maas-api-revoke-keys-e2e-api-a-71c0c7-ptgs7\" (UID: \"d7b824c4-575b-4c1a-bcf4-0f833dd21f70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71c0c7-ptgs7" Oct 30 21:59:00.060234 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:00.060074 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7b824c4-575b-4c1a-bcf4-0f833dd21f70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-71c0c7-ptgs7\" (UID: \"d7b824c4-575b-4c1a-bcf4-0f833dd21f70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71c0c7-ptgs7" Oct 30 21:59:00.060234 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:00.060151 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7b824c4-575b-4c1a-bcf4-0f833dd21f70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-71c0c7-ptgs7\" (UID: \"d7b824c4-575b-4c1a-bcf4-0f833dd21f70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71c0c7-ptgs7" Oct 30 21:59:00.060848 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:00.060823 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7b824c4-575b-4c1a-bcf4-0f833dd21f70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-71c0c7-ptgs7\" (UID: \"d7b824c4-575b-4c1a-bcf4-0f833dd21f70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71c0c7-ptgs7" Oct 30 21:59:00.062445 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:00.062424 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7b824c4-575b-4c1a-bcf4-0f833dd21f70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-71c0c7-ptgs7\" (UID: \"d7b824c4-575b-4c1a-bcf4-0f833dd21f70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71c0c7-ptgs7" Oct 30 21:59:00.120879 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:00.120852 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v6z9\" (UniqueName: \"kubernetes.io/projected/d7b824c4-575b-4c1a-bcf4-0f833dd21f70-kube-api-access-9v6z9\") pod \"maas-api-revoke-keys-e2e-api-a-71c0c7-ptgs7\" (UID: \"d7b824c4-575b-4c1a-bcf4-0f833dd21f70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71c0c7-ptgs7" Oct 30 21:59:00.211005 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:00.210928 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71c0c7-ptgs7" Oct 30 21:59:00.591487 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:00.591460 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71c0c7-ptgs7"] Oct 30 21:59:00.596432 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:59:00.596406 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7b824c4_575b_4c1a_bcf4_0f833dd21f70.slice/crio-6528016baab6b276319b6a99e98a9e3ac3ef1babdcbaa35d168ea09c39506492 WatchSource:0}: Error finding container 6528016baab6b276319b6a99e98a9e3ac3ef1babdcbaa35d168ea09c39506492: Status 404 returned error can't find the container with id 6528016baab6b276319b6a99e98a9e3ac3ef1babdcbaa35d168ea09c39506492 Oct 30 21:59:00.954498 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:00.954463 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71c0c7-ptgs7" event={"ID":"d7b824c4-575b-4c1a-bcf4-0f833dd21f70","Type":"ContainerStarted","Data":"94b2c9d4ed0f0badd36f6220afba88e75d356f0541f8fa5fff13c9c54bf49fe2"} Oct 30 21:59:00.954944 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:00.954518 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71c0c7-ptgs7" event={"ID":"d7b824c4-575b-4c1a-bcf4-0f833dd21f70","Type":"ContainerStarted","Data":"6528016baab6b276319b6a99e98a9e3ac3ef1babdcbaa35d168ea09c39506492"} Oct 30 21:59:01.066755 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:01.066702 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71c0c7-ptgs7" podStartSLOduration=2.066683827 podStartE2EDuration="2.066683827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:01.065421157 +0000 UTC m=+2716.620120934" watchObservedRunningTime="2026-10-30 21:59:01.066683827 +0000 UTC m=+2716.621383640" Oct 30 21:59:01.961759 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:01.961669 4560 generic.go:356] "Generic (PLEG): container finished" podID="d7b824c4-575b-4c1a-bcf4-0f833dd21f70" containerID="94b2c9d4ed0f0badd36f6220afba88e75d356f0541f8fa5fff13c9c54bf49fe2" exitCode=0 Oct 30 21:59:01.962135 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:01.961770 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71c0c7-ptgs7" event={"ID":"d7b824c4-575b-4c1a-bcf4-0f833dd21f70","Type":"ContainerDied","Data":"94b2c9d4ed0f0badd36f6220afba88e75d356f0541f8fa5fff13c9c54bf49fe2"} Oct 30 21:59:03.129106 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:03.129069 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71c0c7-ptgs7" Oct 30 21:59:03.194948 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:03.194915 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7b824c4-575b-4c1a-bcf4-0f833dd21f70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7b824c4-575b-4c1a-bcf4-0f833dd21f70-maas-api-openshift-ca\") pod \"d7b824c4-575b-4c1a-bcf4-0f833dd21f70\" (UID: \"d7b824c4-575b-4c1a-bcf4-0f833dd21f70\") " Oct 30 21:59:03.195171 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:03.194987 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7b824c4-575b-4c1a-bcf4-0f833dd21f70-kube-api-access-9v6z9\" (UniqueName: \"kubernetes.io/projected/d7b824c4-575b-4c1a-bcf4-0f833dd21f70-kube-api-access-9v6z9\") pod \"d7b824c4-575b-4c1a-bcf4-0f833dd21f70\" (UID: \"d7b824c4-575b-4c1a-bcf4-0f833dd21f70\") " Oct 30 21:59:03.195171 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:03.195022 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7b824c4-575b-4c1a-bcf4-0f833dd21f70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7b824c4-575b-4c1a-bcf4-0f833dd21f70-maas-api-xks-ca\") pod \"d7b824c4-575b-4c1a-bcf4-0f833dd21f70\" (UID: \"d7b824c4-575b-4c1a-bcf4-0f833dd21f70\") " Oct 30 21:59:03.195329 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:03.195303 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7b824c4-575b-4c1a-bcf4-0f833dd21f70-maas-api-openshift-ca" pod "d7b824c4-575b-4c1a-bcf4-0f833dd21f70" (UID: "d7b824c4-575b-4c1a-bcf4-0f833dd21f70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:03.195520 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:03.195501 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7b824c4-575b-4c1a-bcf4-0f833dd21f70-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:03.197519 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:03.197489 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7b824c4-575b-4c1a-bcf4-0f833dd21f70-maas-api-xks-ca" pod "d7b824c4-575b-4c1a-bcf4-0f833dd21f70" (UID: "d7b824c4-575b-4c1a-bcf4-0f833dd21f70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:03.197904 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:03.197885 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7b824c4-575b-4c1a-bcf4-0f833dd21f70-kube-api-access-9v6z9" pod "d7b824c4-575b-4c1a-bcf4-0f833dd21f70" (UID: "d7b824c4-575b-4c1a-bcf4-0f833dd21f70"). InnerVolumeSpecName "kube-api-access-9v6z9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:03.309364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:03.297046 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v6z9\" (UniqueName: \"kubernetes.io/projected/d7b824c4-575b-4c1a-bcf4-0f833dd21f70-kube-api-access-9v6z9\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:03.309364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:03.297086 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7b824c4-575b-4c1a-bcf4-0f833dd21f70-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:03.978846 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:03.978805 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71c0c7-ptgs7" event={"ID":"d7b824c4-575b-4c1a-bcf4-0f833dd21f70","Type":"ContainerDied","Data":"6528016baab6b276319b6a99e98a9e3ac3ef1babdcbaa35d168ea09c39506492"} Oct 30 21:59:03.979046 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:03.978817 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71c0c7-ptgs7" Oct 30 21:59:03.979046 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:03.978882 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6528016baab6b276319b6a99e98a9e3ac3ef1babdcbaa35d168ea09c39506492" Oct 30 21:59:19.685532 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:59:19.685492 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:59:23.902849 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:23.902815 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74fbce9f-qlm7g"] Oct 30 21:59:23.903443 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:23.903419 4560 cpu_manager.go:414] "containerMap: removing container" podUID="d7b824c4-575b-4c1a-bcf4-0f833dd21f70" containerName="revoke-keys" Oct 30 21:59:23.903443 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:23.903440 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7b824c4-575b-4c1a-bcf4-0f833dd21f70" containerName="revoke-keys" Oct 30 21:59:23.903622 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:23.903463 4560 cpu_manager.go:414] "containerMap: removing container" podUID="7a8c5ad5-8b50-4795-9596-d951de399015" containerName="istio-proxy" Oct 30 21:59:23.903622 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:23.903471 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a8c5ad5-8b50-4795-9596-d951de399015" containerName="istio-proxy" Oct 30 21:59:23.903622 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:23.903606 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7b824c4-575b-4c1a-bcf4-0f833dd21f70" containerName="revoke-keys" Oct 30 21:59:23.906875 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:23.906860 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fbce9f-qlm7g" Oct 30 21:59:23.960249 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:23.960224 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:59:23.960414 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:23.960368 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:59:23.964729 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:23.964704 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74fbce9f-qlm7g"] Oct 30 21:59:24.005492 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:24.005468 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cbhh\" (UniqueName: \"kubernetes.io/projected/70bcada5-5c22-4367-9743-39a61a86a992-kube-api-access-5cbhh\") pod \"maas-api-revoke-sub-74fbce9f-qlm7g\" (UID: \"70bcada5-5c22-4367-9743-39a61a86a992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fbce9f-qlm7g" Oct 30 21:59:24.005626 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:24.005519 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70bcada5-5c22-4367-9743-39a61a86a992-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74fbce9f-qlm7g\" (UID: \"70bcada5-5c22-4367-9743-39a61a86a992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fbce9f-qlm7g" Oct 30 21:59:24.005750 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:24.005713 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70bcada5-5c22-4367-9743-39a61a86a992-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74fbce9f-qlm7g\" (UID: \"70bcada5-5c22-4367-9743-39a61a86a992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fbce9f-qlm7g" Oct 30 21:59:24.006654 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:24.006638 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:59:24.007100 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:24.007081 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:59:24.010045 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:24.010024 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0a1500a-5fqzs"] Oct 30 21:59:24.013672 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:24.013657 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a1500a-5fqzs" Oct 30 21:59:24.034844 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:24.034824 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0a1500a-5fqzs"] Oct 30 21:59:24.107163 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:24.107132 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70bcada5-5c22-4367-9743-39a61a86a992-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74fbce9f-qlm7g\" (UID: \"70bcada5-5c22-4367-9743-39a61a86a992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fbce9f-qlm7g" Oct 30 21:59:24.107331 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:24.107233 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cbhh\" (UniqueName: \"kubernetes.io/projected/70bcada5-5c22-4367-9743-39a61a86a992-kube-api-access-5cbhh\") pod \"maas-api-revoke-sub-74fbce9f-qlm7g\" (UID: \"70bcada5-5c22-4367-9743-39a61a86a992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fbce9f-qlm7g" Oct 30 21:59:24.107331 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:24.107257 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f596e63b-dac7-425b-bec4-497f31cb8f26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0a1500a-5fqzs\" (UID: \"f596e63b-dac7-425b-bec4-497f31cb8f26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a1500a-5fqzs" Oct 30 21:59:24.107331 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:24.107281 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70bcada5-5c22-4367-9743-39a61a86a992-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74fbce9f-qlm7g\" (UID: \"70bcada5-5c22-4367-9743-39a61a86a992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fbce9f-qlm7g" Oct 30 21:59:24.107331 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:24.107307 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f596e63b-dac7-425b-bec4-497f31cb8f26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0a1500a-5fqzs\" (UID: \"f596e63b-dac7-425b-bec4-497f31cb8f26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a1500a-5fqzs" Oct 30 21:59:24.107516 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:24.107489 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5bvj\" (UniqueName: \"kubernetes.io/projected/f596e63b-dac7-425b-bec4-497f31cb8f26-kube-api-access-m5bvj\") pod \"maas-api-revoke-sub-f0a1500a-5fqzs\" (UID: \"f596e63b-dac7-425b-bec4-497f31cb8f26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a1500a-5fqzs" Oct 30 21:59:24.107829 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:24.107810 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70bcada5-5c22-4367-9743-39a61a86a992-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74fbce9f-qlm7g\" (UID: \"70bcada5-5c22-4367-9743-39a61a86a992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fbce9f-qlm7g" Oct 30 21:59:24.109665 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:24.109649 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70bcada5-5c22-4367-9743-39a61a86a992-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74fbce9f-qlm7g\" (UID: \"70bcada5-5c22-4367-9743-39a61a86a992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fbce9f-qlm7g" Oct 30 21:59:24.165715 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:24.165639 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cbhh\" (UniqueName: \"kubernetes.io/projected/70bcada5-5c22-4367-9743-39a61a86a992-kube-api-access-5cbhh\") pod \"maas-api-revoke-sub-74fbce9f-qlm7g\" (UID: \"70bcada5-5c22-4367-9743-39a61a86a992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fbce9f-qlm7g" Oct 30 21:59:24.208730 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:24.208694 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5bvj\" (UniqueName: \"kubernetes.io/projected/f596e63b-dac7-425b-bec4-497f31cb8f26-kube-api-access-m5bvj\") pod \"maas-api-revoke-sub-f0a1500a-5fqzs\" (UID: \"f596e63b-dac7-425b-bec4-497f31cb8f26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a1500a-5fqzs" Oct 30 21:59:24.208921 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:24.208875 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f596e63b-dac7-425b-bec4-497f31cb8f26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0a1500a-5fqzs\" (UID: \"f596e63b-dac7-425b-bec4-497f31cb8f26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a1500a-5fqzs" Oct 30 21:59:24.208991 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:24.208927 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f596e63b-dac7-425b-bec4-497f31cb8f26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0a1500a-5fqzs\" (UID: \"f596e63b-dac7-425b-bec4-497f31cb8f26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a1500a-5fqzs" Oct 30 21:59:24.209703 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:24.209677 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f596e63b-dac7-425b-bec4-497f31cb8f26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0a1500a-5fqzs\" (UID: \"f596e63b-dac7-425b-bec4-497f31cb8f26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a1500a-5fqzs" Oct 30 21:59:24.211575 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:24.211555 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f596e63b-dac7-425b-bec4-497f31cb8f26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0a1500a-5fqzs\" (UID: \"f596e63b-dac7-425b-bec4-497f31cb8f26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a1500a-5fqzs" Oct 30 21:59:24.219527 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:24.219506 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fbce9f-qlm7g" Oct 30 21:59:24.266901 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:24.266873 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5bvj\" (UniqueName: \"kubernetes.io/projected/f596e63b-dac7-425b-bec4-497f31cb8f26-kube-api-access-m5bvj\") pod \"maas-api-revoke-sub-f0a1500a-5fqzs\" (UID: \"f596e63b-dac7-425b-bec4-497f31cb8f26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a1500a-5fqzs" Oct 30 21:59:24.326770 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:24.326738 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a1500a-5fqzs" Oct 30 21:59:24.595717 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:24.595687 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74fbce9f-qlm7g"] Oct 30 21:59:24.601480 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:59:24.601450 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70bcada5_5c22_4367_9743_39a61a86a992.slice/crio-9e63d60ed854189bd6ad670ab2bb85220ce6dfbdaa9406d31eff9ed9bf63df1e WatchSource:0}: Error finding container 9e63d60ed854189bd6ad670ab2bb85220ce6dfbdaa9406d31eff9ed9bf63df1e: Status 404 returned error can't find the container with id 9e63d60ed854189bd6ad670ab2bb85220ce6dfbdaa9406d31eff9ed9bf63df1e Oct 30 21:59:24.710812 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:24.710782 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0a1500a-5fqzs"] Oct 30 21:59:24.716044 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:59:24.716020 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf596e63b_dac7_425b_bec4_497f31cb8f26.slice/crio-19f24129a471831599654180f2a905279676e04c6cb724a535254ee95a08c8f6 WatchSource:0}: Error finding container 19f24129a471831599654180f2a905279676e04c6cb724a535254ee95a08c8f6: Status 404 returned error can't find the container with id 19f24129a471831599654180f2a905279676e04c6cb724a535254ee95a08c8f6 Oct 30 21:59:24.798495 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:24.794108 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-27114d-chj7h"] Oct 30 21:59:24.808517 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:24.808486 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-27114d-chj7h"] Oct 30 21:59:25.135532 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:25.135485 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a1500a-5fqzs" event={"ID":"f596e63b-dac7-425b-bec4-497f31cb8f26","Type":"ContainerStarted","Data":"7f4bf1cabc1afb82734f606818b715f74ec2ff0eba2631cf589a68205ffb4f24"} Oct 30 21:59:25.136071 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:25.135594 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a1500a-5fqzs" event={"ID":"f596e63b-dac7-425b-bec4-497f31cb8f26","Type":"ContainerStarted","Data":"19f24129a471831599654180f2a905279676e04c6cb724a535254ee95a08c8f6"} Oct 30 21:59:25.137800 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:25.137765 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fbce9f-qlm7g" event={"ID":"70bcada5-5c22-4367-9743-39a61a86a992","Type":"ContainerStarted","Data":"b8b11d7dff2712a080062df716af51f2f06862f133dd4a2cdfd164e24d676325"} Oct 30 21:59:25.137935 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:25.137820 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fbce9f-qlm7g" event={"ID":"70bcada5-5c22-4367-9743-39a61a86a992","Type":"ContainerStarted","Data":"9e63d60ed854189bd6ad670ab2bb85220ce6dfbdaa9406d31eff9ed9bf63df1e"} Oct 30 21:59:25.250479 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:25.250425 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a1500a-5fqzs" podStartSLOduration=2.250411295 podStartE2EDuration="2.250411295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:25.247702273 +0000 UTC m=+2740.802402051" watchObservedRunningTime="2026-10-30 21:59:25.250411295 +0000 UTC m=+2740.805111072" Oct 30 21:59:25.358159 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:25.358102 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fbce9f-qlm7g" podStartSLOduration=2.358088125 podStartE2EDuration="2.358088125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:25.357031701 +0000 UTC m=+2740.911731482" watchObservedRunningTime="2026-10-30 21:59:25.358088125 +0000 UTC m=+2740.912787902" Oct 30 21:59:26.145127 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:26.145093 4560 generic.go:356] "Generic (PLEG): container finished" podID="70bcada5-5c22-4367-9743-39a61a86a992" containerID="b8b11d7dff2712a080062df716af51f2f06862f133dd4a2cdfd164e24d676325" exitCode=0 Oct 30 21:59:26.145574 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:26.145181 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fbce9f-qlm7g" event={"ID":"70bcada5-5c22-4367-9743-39a61a86a992","Type":"ContainerDied","Data":"b8b11d7dff2712a080062df716af51f2f06862f133dd4a2cdfd164e24d676325"} Oct 30 21:59:26.146797 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:26.146770 4560 generic.go:356] "Generic (PLEG): container finished" podID="f596e63b-dac7-425b-bec4-497f31cb8f26" containerID="7f4bf1cabc1afb82734f606818b715f74ec2ff0eba2631cf589a68205ffb4f24" exitCode=0 Oct 30 21:59:26.146913 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:26.146846 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a1500a-5fqzs" event={"ID":"f596e63b-dac7-425b-bec4-497f31cb8f26","Type":"ContainerDied","Data":"7f4bf1cabc1afb82734f606818b715f74ec2ff0eba2631cf589a68205ffb4f24"} Oct 30 21:59:26.576575 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:26.576539 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa9bc5ce-d25b-4010-ac88-48f27a08338c" path="/var/lib/kubelet/pods/aa9bc5ce-d25b-4010-ac88-48f27a08338c/volumes" Oct 30 21:59:27.306144 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:27.306117 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fbce9f-qlm7g" Oct 30 21:59:27.337630 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:27.337604 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a1500a-5fqzs" Oct 30 21:59:27.346797 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:27.346776 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70bcada5-5c22-4367-9743-39a61a86a992-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70bcada5-5c22-4367-9743-39a61a86a992-maas-api-openshift-ca\") pod \"70bcada5-5c22-4367-9743-39a61a86a992\" (UID: \"70bcada5-5c22-4367-9743-39a61a86a992\") " Oct 30 21:59:27.346883 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:27.346865 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70bcada5-5c22-4367-9743-39a61a86a992-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70bcada5-5c22-4367-9743-39a61a86a992-maas-api-xks-ca\") pod \"70bcada5-5c22-4367-9743-39a61a86a992\" (UID: \"70bcada5-5c22-4367-9743-39a61a86a992\") " Oct 30 21:59:27.346926 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:27.346917 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70bcada5-5c22-4367-9743-39a61a86a992-kube-api-access-5cbhh\" (UniqueName: \"kubernetes.io/projected/70bcada5-5c22-4367-9743-39a61a86a992-kube-api-access-5cbhh\") pod \"70bcada5-5c22-4367-9743-39a61a86a992\" (UID: \"70bcada5-5c22-4367-9743-39a61a86a992\") " Oct 30 21:59:27.347181 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:27.347149 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70bcada5-5c22-4367-9743-39a61a86a992-maas-api-openshift-ca" pod "70bcada5-5c22-4367-9743-39a61a86a992" (UID: "70bcada5-5c22-4367-9743-39a61a86a992"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:27.347302 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:27.347287 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70bcada5-5c22-4367-9743-39a61a86a992-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:27.348908 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:27.348885 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70bcada5-5c22-4367-9743-39a61a86a992-maas-api-xks-ca" pod "70bcada5-5c22-4367-9743-39a61a86a992" (UID: "70bcada5-5c22-4367-9743-39a61a86a992"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:27.348977 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:27.348942 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70bcada5-5c22-4367-9743-39a61a86a992-kube-api-access-5cbhh" pod "70bcada5-5c22-4367-9743-39a61a86a992" (UID: "70bcada5-5c22-4367-9743-39a61a86a992"). InnerVolumeSpecName "kube-api-access-5cbhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:27.447776 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:27.447696 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f596e63b-dac7-425b-bec4-497f31cb8f26-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f596e63b-dac7-425b-bec4-497f31cb8f26-maas-api-openshift-ca\") pod \"f596e63b-dac7-425b-bec4-497f31cb8f26\" (UID: \"f596e63b-dac7-425b-bec4-497f31cb8f26\") " Oct 30 21:59:27.447776 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:27.447765 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f596e63b-dac7-425b-bec4-497f31cb8f26-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f596e63b-dac7-425b-bec4-497f31cb8f26-maas-api-xks-ca\") pod \"f596e63b-dac7-425b-bec4-497f31cb8f26\" (UID: \"f596e63b-dac7-425b-bec4-497f31cb8f26\") " Oct 30 21:59:27.448008 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:27.447951 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f596e63b-dac7-425b-bec4-497f31cb8f26-kube-api-access-m5bvj\" (UniqueName: \"kubernetes.io/projected/f596e63b-dac7-425b-bec4-497f31cb8f26-kube-api-access-m5bvj\") pod \"f596e63b-dac7-425b-bec4-497f31cb8f26\" (UID: \"f596e63b-dac7-425b-bec4-497f31cb8f26\") " Oct 30 21:59:27.448124 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:27.448091 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f596e63b-dac7-425b-bec4-497f31cb8f26-maas-api-openshift-ca" pod "f596e63b-dac7-425b-bec4-497f31cb8f26" (UID: "f596e63b-dac7-425b-bec4-497f31cb8f26"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:27.448411 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:27.448389 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f596e63b-dac7-425b-bec4-497f31cb8f26-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:27.448411 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:27.448409 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70bcada5-5c22-4367-9743-39a61a86a992-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:27.448604 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:27.448424 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cbhh\" (UniqueName: \"kubernetes.io/projected/70bcada5-5c22-4367-9743-39a61a86a992-kube-api-access-5cbhh\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:27.450433 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:27.450412 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f596e63b-dac7-425b-bec4-497f31cb8f26-maas-api-xks-ca" pod "f596e63b-dac7-425b-bec4-497f31cb8f26" (UID: "f596e63b-dac7-425b-bec4-497f31cb8f26"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:27.450543 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:27.450445 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f596e63b-dac7-425b-bec4-497f31cb8f26-kube-api-access-m5bvj" pod "f596e63b-dac7-425b-bec4-497f31cb8f26" (UID: "f596e63b-dac7-425b-bec4-497f31cb8f26"). InnerVolumeSpecName "kube-api-access-m5bvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:27.549504 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:27.549469 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5bvj\" (UniqueName: \"kubernetes.io/projected/f596e63b-dac7-425b-bec4-497f31cb8f26-kube-api-access-m5bvj\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:27.549504 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:27.549499 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f596e63b-dac7-425b-bec4-497f31cb8f26-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:28.160942 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:28.160915 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fbce9f-qlm7g" Oct 30 21:59:28.161153 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:28.160960 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fbce9f-qlm7g" event={"ID":"70bcada5-5c22-4367-9743-39a61a86a992","Type":"ContainerDied","Data":"9e63d60ed854189bd6ad670ab2bb85220ce6dfbdaa9406d31eff9ed9bf63df1e"} Oct 30 21:59:28.161153 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:28.161003 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e63d60ed854189bd6ad670ab2bb85220ce6dfbdaa9406d31eff9ed9bf63df1e" Oct 30 21:59:28.162374 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:28.162323 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a1500a-5fqzs" event={"ID":"f596e63b-dac7-425b-bec4-497f31cb8f26","Type":"ContainerDied","Data":"19f24129a471831599654180f2a905279676e04c6cb724a535254ee95a08c8f6"} Oct 30 21:59:28.162500 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:28.162381 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a1500a-5fqzs" Oct 30 21:59:28.162500 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:28.162387 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19f24129a471831599654180f2a905279676e04c6cb724a535254ee95a08c8f6" Oct 30 21:59:33.849658 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:33.849625 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ed6135-4hx4s"] Oct 30 21:59:33.850898 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:33.850873 4560 cpu_manager.go:414] "containerMap: removing container" podUID="70bcada5-5c22-4367-9743-39a61a86a992" containerName="revoke-keys" Oct 30 21:59:33.850898 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:33.850896 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70bcada5-5c22-4367-9743-39a61a86a992" containerName="revoke-keys" Oct 30 21:59:33.851071 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:33.850938 4560 cpu_manager.go:414] "containerMap: removing container" podUID="f596e63b-dac7-425b-bec4-497f31cb8f26" containerName="revoke-keys" Oct 30 21:59:33.851071 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:33.850945 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f596e63b-dac7-425b-bec4-497f31cb8f26" containerName="revoke-keys" Oct 30 21:59:33.851071 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:33.851066 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="70bcada5-5c22-4367-9743-39a61a86a992" containerName="revoke-keys" Oct 30 21:59:33.851197 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:33.851084 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="f596e63b-dac7-425b-bec4-497f31cb8f26" containerName="revoke-keys" Oct 30 21:59:33.854878 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:33.854860 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ed6135-4hx4s" Oct 30 21:59:33.904712 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:33.904661 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:59:33.909591 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:33.909560 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ed6135-4hx4s"] Oct 30 21:59:33.933090 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:33.933060 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4556ce3a-898e-49ee-8838-3446807183fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ed6135-4hx4s\" (UID: \"4556ce3a-898e-49ee-8838-3446807183fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ed6135-4hx4s" Oct 30 21:59:33.933237 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:33.933131 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvglm\" (UniqueName: \"kubernetes.io/projected/4556ce3a-898e-49ee-8838-3446807183fa-kube-api-access-xvglm\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ed6135-4hx4s\" (UID: \"4556ce3a-898e-49ee-8838-3446807183fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ed6135-4hx4s" Oct 30 21:59:33.933305 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:33.933240 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4556ce3a-898e-49ee-8838-3446807183fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ed6135-4hx4s\" (UID: \"4556ce3a-898e-49ee-8838-3446807183fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ed6135-4hx4s" Oct 30 21:59:33.951550 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:33.951516 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:59:33.951709 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:33.951595 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:59:33.951709 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:33.951668 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:59:34.035199 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:34.035167 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4556ce3a-898e-49ee-8838-3446807183fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ed6135-4hx4s\" (UID: \"4556ce3a-898e-49ee-8838-3446807183fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ed6135-4hx4s" Oct 30 21:59:34.035324 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:34.035229 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvglm\" (UniqueName: \"kubernetes.io/projected/4556ce3a-898e-49ee-8838-3446807183fa-kube-api-access-xvglm\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ed6135-4hx4s\" (UID: \"4556ce3a-898e-49ee-8838-3446807183fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ed6135-4hx4s" Oct 30 21:59:34.035324 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:34.035257 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4556ce3a-898e-49ee-8838-3446807183fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ed6135-4hx4s\" (UID: \"4556ce3a-898e-49ee-8838-3446807183fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ed6135-4hx4s" Oct 30 21:59:34.035899 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:34.035881 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4556ce3a-898e-49ee-8838-3446807183fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ed6135-4hx4s\" (UID: \"4556ce3a-898e-49ee-8838-3446807183fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ed6135-4hx4s" Oct 30 21:59:34.037472 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:34.037455 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4556ce3a-898e-49ee-8838-3446807183fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ed6135-4hx4s\" (UID: \"4556ce3a-898e-49ee-8838-3446807183fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ed6135-4hx4s" Oct 30 21:59:34.095172 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:34.095144 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvglm\" (UniqueName: \"kubernetes.io/projected/4556ce3a-898e-49ee-8838-3446807183fa-kube-api-access-xvglm\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ed6135-4hx4s\" (UID: \"4556ce3a-898e-49ee-8838-3446807183fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ed6135-4hx4s" Oct 30 21:59:34.167315 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:34.167241 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ed6135-4hx4s" Oct 30 21:59:34.544455 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:34.544420 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ed6135-4hx4s"] Oct 30 21:59:34.549191 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:59:34.549165 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4556ce3a_898e_49ee_8838_3446807183fa.slice/crio-a71b3f3f671536f72d245ba6535983f03e0aba500dfc571954353d903b39fcac WatchSource:0}: Error finding container a71b3f3f671536f72d245ba6535983f03e0aba500dfc571954353d903b39fcac: Status 404 returned error can't find the container with id a71b3f3f671536f72d245ba6535983f03e0aba500dfc571954353d903b39fcac Oct 30 21:59:35.205927 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:35.205839 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ed6135-4hx4s" event={"ID":"4556ce3a-898e-49ee-8838-3446807183fa","Type":"ContainerStarted","Data":"1e41a2699cba177342cfd9a7ea0aef1b76bd8a4e34c342db72ecaa63cb435404"} Oct 30 21:59:35.205927 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:35.205902 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ed6135-4hx4s" event={"ID":"4556ce3a-898e-49ee-8838-3446807183fa","Type":"ContainerStarted","Data":"a71b3f3f671536f72d245ba6535983f03e0aba500dfc571954353d903b39fcac"} Oct 30 21:59:35.316456 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:35.316403 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ed6135-4hx4s" podStartSLOduration=2.316390766 podStartE2EDuration="2.316390766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:35.315507772 +0000 UTC m=+2750.870207550" watchObservedRunningTime="2026-10-30 21:59:35.316390766 +0000 UTC m=+2750.871090543" Oct 30 21:59:35.979095 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:35.979060 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 21:59:36.213403 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:36.213359 4560 generic.go:356] "Generic (PLEG): container finished" podID="4556ce3a-898e-49ee-8838-3446807183fa" containerID="1e41a2699cba177342cfd9a7ea0aef1b76bd8a4e34c342db72ecaa63cb435404" exitCode=0 Oct 30 21:59:36.213827 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:36.213442 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ed6135-4hx4s" event={"ID":"4556ce3a-898e-49ee-8838-3446807183fa","Type":"ContainerDied","Data":"1e41a2699cba177342cfd9a7ea0aef1b76bd8a4e34c342db72ecaa63cb435404"} Oct 30 21:59:37.499025 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:37.498998 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ed6135-4hx4s" Oct 30 21:59:37.522283 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:37.522254 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4556ce3a-898e-49ee-8838-3446807183fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4556ce3a-898e-49ee-8838-3446807183fa-maas-api-openshift-ca\") pod \"4556ce3a-898e-49ee-8838-3446807183fa\" (UID: \"4556ce3a-898e-49ee-8838-3446807183fa\") " Oct 30 21:59:37.522463 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:37.522447 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4556ce3a-898e-49ee-8838-3446807183fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4556ce3a-898e-49ee-8838-3446807183fa-maas-api-xks-ca\") pod \"4556ce3a-898e-49ee-8838-3446807183fa\" (UID: \"4556ce3a-898e-49ee-8838-3446807183fa\") " Oct 30 21:59:37.522539 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:37.522491 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4556ce3a-898e-49ee-8838-3446807183fa-kube-api-access-xvglm\" (UniqueName: \"kubernetes.io/projected/4556ce3a-898e-49ee-8838-3446807183fa-kube-api-access-xvglm\") pod \"4556ce3a-898e-49ee-8838-3446807183fa\" (UID: \"4556ce3a-898e-49ee-8838-3446807183fa\") " Oct 30 21:59:37.522657 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:37.522630 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4556ce3a-898e-49ee-8838-3446807183fa-maas-api-openshift-ca" pod "4556ce3a-898e-49ee-8838-3446807183fa" (UID: "4556ce3a-898e-49ee-8838-3446807183fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:37.522923 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:37.522903 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4556ce3a-898e-49ee-8838-3446807183fa-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:37.524491 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:37.524467 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4556ce3a-898e-49ee-8838-3446807183fa-maas-api-xks-ca" pod "4556ce3a-898e-49ee-8838-3446807183fa" (UID: "4556ce3a-898e-49ee-8838-3446807183fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:37.524579 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:37.524473 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4556ce3a-898e-49ee-8838-3446807183fa-kube-api-access-xvglm" pod "4556ce3a-898e-49ee-8838-3446807183fa" (UID: "4556ce3a-898e-49ee-8838-3446807183fa"). InnerVolumeSpecName "kube-api-access-xvglm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:37.623655 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:37.623624 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4556ce3a-898e-49ee-8838-3446807183fa-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:37.623655 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:37.623653 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvglm\" (UniqueName: \"kubernetes.io/projected/4556ce3a-898e-49ee-8838-3446807183fa-kube-api-access-xvglm\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:38.230727 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:38.230691 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ed6135-4hx4s" event={"ID":"4556ce3a-898e-49ee-8838-3446807183fa","Type":"ContainerDied","Data":"a71b3f3f671536f72d245ba6535983f03e0aba500dfc571954353d903b39fcac"} Oct 30 21:59:38.230899 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:38.230746 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a71b3f3f671536f72d245ba6535983f03e0aba500dfc571954353d903b39fcac" Oct 30 21:59:38.230899 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:38.230706 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ed6135-4hx4s" Oct 30 21:59:38.466935 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:38.466897 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h"] Oct 30 21:59:38.467201 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:38.467143 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" podUID="e71b8a5c-fcd2-44f0-86d4-984ab9512afd" containerName="istio-proxy" containerID="cri-o://000119b73bc1afe20c21ac31375dae42bc0e428028d3389b510571d8568929e7" gracePeriod=30 Oct 30 21:59:39.949409 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:39.949366 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn"] Oct 30 21:59:39.949825 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:39.949563 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" podUID="d64f50e0-cf7a-43ec-83ac-a73c0972cb39" containerName="istio-proxy" containerID="cri-o://53e8673f6d93be057b26cf7786f8e4a7dde2460b4e1389dd43746eb2799934d4" gracePeriod=30 Oct 30 21:59:40.366632 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:40.366597 4560 patch_prober.go:29] interesting pod/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:59:40.366820 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:40.366671 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" podUID="e71b8a5c-fcd2-44f0-86d4-984ab9512afd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:59:41.417403 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:41.417363 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-737a64-htl7r"] Oct 30 21:59:41.464042 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:41.464011 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-737a64-htl7r"] Oct 30 21:59:42.240482 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:42.240446 4560 patch_prober.go:29] interesting pod/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:59:42.240655 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:42.240502 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" podUID="d64f50e0-cf7a-43ec-83ac-a73c0972cb39" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:59:42.576221 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:42.576130 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49cb202f-c2ed-4107-89e6-dd5c5adef978" path="/var/lib/kubelet/pods/49cb202f-c2ed-4107-89e6-dd5c5adef978/volumes" Oct 30 21:59:43.731635 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:43.731580 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" Oct 30 21:59:43.818058 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:43.818027 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-istio-podinfo\") pod \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\" (UID: \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\") " Oct 30 21:59:43.818227 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:43.818066 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-istio-envoy\") pod \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\" (UID: \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\") " Oct 30 21:59:43.818227 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:43.818101 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-istio-data\") pod \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\" (UID: \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\") " Oct 30 21:59:43.818227 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:43.818132 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-kube-api-access-txkwj\" (UniqueName: \"kubernetes.io/projected/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-kube-api-access-txkwj\") pod \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\" (UID: \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\") " Oct 30 21:59:43.818227 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:43.818152 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-istio-token\" (UniqueName: \"kubernetes.io/projected/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-istio-token\") pod \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\" (UID: \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\") " Oct 30 21:59:43.818227 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:43.818179 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-credential-socket\") pod \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\" (UID: \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\") " Oct 30 21:59:43.818561 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:43.818274 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-workload-socket\") pod \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\" (UID: \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\") " Oct 30 21:59:43.818561 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:43.818325 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-istiod-ca-cert\") pod \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\" (UID: \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\") " Oct 30 21:59:43.818561 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:43.818462 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-workload-certs\") pod \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\" (UID: \"e71b8a5c-fcd2-44f0-86d4-984ab9512afd\") " Oct 30 21:59:43.818561 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:43.818519 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-credential-socket" pod "e71b8a5c-fcd2-44f0-86d4-984ab9512afd" (UID: "e71b8a5c-fcd2-44f0-86d4-984ab9512afd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:43.818782 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:43.818654 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-workload-socket" pod "e71b8a5c-fcd2-44f0-86d4-984ab9512afd" (UID: "e71b8a5c-fcd2-44f0-86d4-984ab9512afd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:43.818782 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:43.818770 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-istio-data" pod "e71b8a5c-fcd2-44f0-86d4-984ab9512afd" (UID: "e71b8a5c-fcd2-44f0-86d4-984ab9512afd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:43.818892 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:43.818865 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-workload-certs" pod "e71b8a5c-fcd2-44f0-86d4-984ab9512afd" (UID: "e71b8a5c-fcd2-44f0-86d4-984ab9512afd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:43.818947 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:43.818883 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-istiod-ca-cert" pod "e71b8a5c-fcd2-44f0-86d4-984ab9512afd" (UID: "e71b8a5c-fcd2-44f0-86d4-984ab9512afd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:43.819158 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:43.819135 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-istio-data\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:43.819228 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:43.819164 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-credential-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:43.819228 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:43.819182 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-workload-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:43.819228 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:43.819196 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-istiod-ca-cert\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:43.819228 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:43.819210 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-workload-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:43.820496 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:43.820466 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-istio-podinfo" pod "e71b8a5c-fcd2-44f0-86d4-984ab9512afd" (UID: "e71b8a5c-fcd2-44f0-86d4-984ab9512afd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:59:43.820496 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:43.820472 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-istio-token" pod "e71b8a5c-fcd2-44f0-86d4-984ab9512afd" (UID: "e71b8a5c-fcd2-44f0-86d4-984ab9512afd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:43.820496 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:43.820490 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-kube-api-access-txkwj" pod "e71b8a5c-fcd2-44f0-86d4-984ab9512afd" (UID: "e71b8a5c-fcd2-44f0-86d4-984ab9512afd"). InnerVolumeSpecName "kube-api-access-txkwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:43.820719 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:43.820493 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-istio-envoy" pod "e71b8a5c-fcd2-44f0-86d4-984ab9512afd" (UID: "e71b8a5c-fcd2-44f0-86d4-984ab9512afd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:43.919946 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:43.919917 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-istio-podinfo\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:43.919946 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:43.919942 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-istio-envoy\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:43.919946 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:43.919952 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txkwj\" (UniqueName: \"kubernetes.io/projected/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-kube-api-access-txkwj\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:43.920162 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:43.919963 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e71b8a5c-fcd2-44f0-86d4-984ab9512afd-istio-token\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:44.283311 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:44.283278 4560 generic.go:356] "Generic (PLEG): container finished" podID="e71b8a5c-fcd2-44f0-86d4-984ab9512afd" containerID="000119b73bc1afe20c21ac31375dae42bc0e428028d3389b510571d8568929e7" exitCode=0 Oct 30 21:59:44.283492 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:44.283326 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" Oct 30 21:59:44.283492 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:44.283370 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" event={"ID":"e71b8a5c-fcd2-44f0-86d4-984ab9512afd","Type":"ContainerDied","Data":"000119b73bc1afe20c21ac31375dae42bc0e428028d3389b510571d8568929e7"} Oct 30 21:59:44.283492 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:44.283426 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h" event={"ID":"e71b8a5c-fcd2-44f0-86d4-984ab9512afd","Type":"ContainerDied","Data":"0c326e63d01ee847d2bfd83aee127cb7972194703f13d276ac7b7485f109b4bb"} Oct 30 21:59:44.283492 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:44.283453 4560 scope.go:122] "RemoveContainer" containerID="000119b73bc1afe20c21ac31375dae42bc0e428028d3389b510571d8568929e7" Oct 30 21:59:44.293793 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:44.293770 4560 scope.go:122] "RemoveContainer" containerID="000119b73bc1afe20c21ac31375dae42bc0e428028d3389b510571d8568929e7" Oct 30 21:59:44.294055 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:59:44.294038 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"000119b73bc1afe20c21ac31375dae42bc0e428028d3389b510571d8568929e7\": container with ID starting with 000119b73bc1afe20c21ac31375dae42bc0e428028d3389b510571d8568929e7 not found: ID does not exist" containerID="000119b73bc1afe20c21ac31375dae42bc0e428028d3389b510571d8568929e7" Oct 30 21:59:44.294105 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:44.294063 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"000119b73bc1afe20c21ac31375dae42bc0e428028d3389b510571d8568929e7"} err="failed to get container status \"000119b73bc1afe20c21ac31375dae42bc0e428028d3389b510571d8568929e7\": rpc error: code = NotFound desc = could not find container \"000119b73bc1afe20c21ac31375dae42bc0e428028d3389b510571d8568929e7\": container with ID starting with 000119b73bc1afe20c21ac31375dae42bc0e428028d3389b510571d8568929e7 not found: ID does not exist" Oct 30 21:59:44.440458 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:44.440418 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h"] Oct 30 21:59:44.487858 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:44.487826 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-54r2h"] Oct 30 21:59:44.578264 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:44.578193 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e71b8a5c-fcd2-44f0-86d4-984ab9512afd" path="/var/lib/kubelet/pods/e71b8a5c-fcd2-44f0-86d4-984ab9512afd/volumes" Oct 30 21:59:45.215622 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:45.215596 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" Oct 30 21:59:45.238072 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:45.238044 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-istiod-ca-cert\") pod \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\" (UID: \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\") " Oct 30 21:59:45.238238 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:45.238175 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-istio-token\" (UniqueName: \"kubernetes.io/projected/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-istio-token\") pod \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\" (UID: \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\") " Oct 30 21:59:45.238238 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:45.238211 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-workload-socket\") pod \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\" (UID: \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\") " Oct 30 21:59:45.238387 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:45.238362 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-istio-envoy\") pod \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\" (UID: \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\") " Oct 30 21:59:45.238458 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:45.238403 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-istiod-ca-cert" pod "d64f50e0-cf7a-43ec-83ac-a73c0972cb39" (UID: "d64f50e0-cf7a-43ec-83ac-a73c0972cb39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:45.238521 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:45.238458 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-istio-podinfo\") pod \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\" (UID: \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\") " Oct 30 21:59:45.238521 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:45.238484 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-kube-api-access-2gjn7\" (UniqueName: \"kubernetes.io/projected/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-kube-api-access-2gjn7\") pod \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\" (UID: \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\") " Oct 30 21:59:45.238521 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:45.238510 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-credential-socket\") pod \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\" (UID: \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\") " Oct 30 21:59:45.238700 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:45.238516 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-workload-socket" pod "d64f50e0-cf7a-43ec-83ac-a73c0972cb39" (UID: "d64f50e0-cf7a-43ec-83ac-a73c0972cb39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:45.238700 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:45.238547 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-workload-certs\") pod \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\" (UID: \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\") " Oct 30 21:59:45.238700 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:45.238633 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-istio-data\") pod \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\" (UID: \"d64f50e0-cf7a-43ec-83ac-a73c0972cb39\") " Oct 30 21:59:45.238865 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:45.238744 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-credential-socket" pod "d64f50e0-cf7a-43ec-83ac-a73c0972cb39" (UID: "d64f50e0-cf7a-43ec-83ac-a73c0972cb39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:45.238947 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:45.238920 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-workload-certs" pod "d64f50e0-cf7a-43ec-83ac-a73c0972cb39" (UID: "d64f50e0-cf7a-43ec-83ac-a73c0972cb39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:45.239110 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:45.239056 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-istio-data" pod "d64f50e0-cf7a-43ec-83ac-a73c0972cb39" (UID: "d64f50e0-cf7a-43ec-83ac-a73c0972cb39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:45.239246 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:45.239230 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-istio-data\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:45.239320 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:45.239251 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-istiod-ca-cert\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:45.239320 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:45.239267 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-workload-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:45.239320 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:45.239283 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-credential-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:45.239320 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:45.239297 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-workload-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:45.240593 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:45.240566 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-istio-podinfo" pod "d64f50e0-cf7a-43ec-83ac-a73c0972cb39" (UID: "d64f50e0-cf7a-43ec-83ac-a73c0972cb39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:59:45.240593 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:45.240574 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-kube-api-access-2gjn7" pod "d64f50e0-cf7a-43ec-83ac-a73c0972cb39" (UID: "d64f50e0-cf7a-43ec-83ac-a73c0972cb39"). InnerVolumeSpecName "kube-api-access-2gjn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:45.240865 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:45.240840 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-istio-token" pod "d64f50e0-cf7a-43ec-83ac-a73c0972cb39" (UID: "d64f50e0-cf7a-43ec-83ac-a73c0972cb39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:45.240865 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:45.240840 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-istio-envoy" pod "d64f50e0-cf7a-43ec-83ac-a73c0972cb39" (UID: "d64f50e0-cf7a-43ec-83ac-a73c0972cb39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:45.295573 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:45.294990 4560 generic.go:356] "Generic (PLEG): container finished" podID="d64f50e0-cf7a-43ec-83ac-a73c0972cb39" containerID="53e8673f6d93be057b26cf7786f8e4a7dde2460b4e1389dd43746eb2799934d4" exitCode=0 Oct 30 21:59:45.295573 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:45.295078 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" event={"ID":"d64f50e0-cf7a-43ec-83ac-a73c0972cb39","Type":"ContainerDied","Data":"53e8673f6d93be057b26cf7786f8e4a7dde2460b4e1389dd43746eb2799934d4"} Oct 30 21:59:45.295573 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:45.295160 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" event={"ID":"d64f50e0-cf7a-43ec-83ac-a73c0972cb39","Type":"ContainerDied","Data":"1e14a284625adb5c924d96b024e79e13add413be7bd75a4f221f694b925f98dd"} Oct 30 21:59:45.295573 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:45.295202 4560 scope.go:122] "RemoveContainer" containerID="53e8673f6d93be057b26cf7786f8e4a7dde2460b4e1389dd43746eb2799934d4" Oct 30 21:59:45.295573 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:45.295335 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn" Oct 30 21:59:45.309192 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:45.309167 4560 scope.go:122] "RemoveContainer" containerID="53e8673f6d93be057b26cf7786f8e4a7dde2460b4e1389dd43746eb2799934d4" Oct 30 21:59:45.309470 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:59:45.309450 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53e8673f6d93be057b26cf7786f8e4a7dde2460b4e1389dd43746eb2799934d4\": container with ID starting with 53e8673f6d93be057b26cf7786f8e4a7dde2460b4e1389dd43746eb2799934d4 not found: ID does not exist" containerID="53e8673f6d93be057b26cf7786f8e4a7dde2460b4e1389dd43746eb2799934d4" Oct 30 21:59:45.309518 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:45.309479 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53e8673f6d93be057b26cf7786f8e4a7dde2460b4e1389dd43746eb2799934d4"} err="failed to get container status \"53e8673f6d93be057b26cf7786f8e4a7dde2460b4e1389dd43746eb2799934d4\": rpc error: code = NotFound desc = could not find container \"53e8673f6d93be057b26cf7786f8e4a7dde2460b4e1389dd43746eb2799934d4\": container with ID starting with 53e8673f6d93be057b26cf7786f8e4a7dde2460b4e1389dd43746eb2799934d4 not found: ID does not exist" Oct 30 21:59:45.340723 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:45.340695 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-istio-envoy\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:45.340861 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:45.340750 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-istio-podinfo\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:45.340861 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:45.340768 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gjn7\" (UniqueName: \"kubernetes.io/projected/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-kube-api-access-2gjn7\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:45.340861 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:45.340783 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d64f50e0-cf7a-43ec-83ac-a73c0972cb39-istio-token\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:45.437707 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:45.437676 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn"] Oct 30 21:59:45.449197 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:45.449165 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-2r5rn"] Oct 30 21:59:46.575321 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:46.575286 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d64f50e0-cf7a-43ec-83ac-a73c0972cb39" path="/var/lib/kubelet/pods/d64f50e0-cf7a-43ec-83ac-a73c0972cb39/volumes" Oct 30 21:59:49.684630 ip-10-0-129-83 kubenswrapper[4560]: E1030 21:59:49.684589 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 21:59:50.057044 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:50.057011 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2df7b6-n2f5r"] Oct 30 21:59:50.057499 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:50.057485 4560 cpu_manager.go:414] "containerMap: removing container" podUID="d64f50e0-cf7a-43ec-83ac-a73c0972cb39" containerName="istio-proxy" Oct 30 21:59:50.057554 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:50.057500 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d64f50e0-cf7a-43ec-83ac-a73c0972cb39" containerName="istio-proxy" Oct 30 21:59:50.057554 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:50.057509 4560 cpu_manager.go:414] "containerMap: removing container" podUID="e71b8a5c-fcd2-44f0-86d4-984ab9512afd" containerName="istio-proxy" Oct 30 21:59:50.057554 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:50.057514 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e71b8a5c-fcd2-44f0-86d4-984ab9512afd" containerName="istio-proxy" Oct 30 21:59:50.057554 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:50.057534 4560 cpu_manager.go:414] "containerMap: removing container" podUID="4556ce3a-898e-49ee-8838-3446807183fa" containerName="revoke-keys" Oct 30 21:59:50.057554 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:50.057539 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4556ce3a-898e-49ee-8838-3446807183fa" containerName="revoke-keys" Oct 30 21:59:50.057710 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:50.057611 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="d64f50e0-cf7a-43ec-83ac-a73c0972cb39" containerName="istio-proxy" Oct 30 21:59:50.057710 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:50.057620 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="e71b8a5c-fcd2-44f0-86d4-984ab9512afd" containerName="istio-proxy" Oct 30 21:59:50.057710 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:50.057629 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="4556ce3a-898e-49ee-8838-3446807183fa" containerName="revoke-keys" Oct 30 21:59:50.062414 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:50.062397 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2df7b6-n2f5r" Oct 30 21:59:50.111980 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:50.111956 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:59:50.158804 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:50.158774 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:59:50.158804 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:50.158812 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:59:50.159026 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:50.158870 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 21:59:50.161748 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:50.161725 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2df7b6-n2f5r"] Oct 30 21:59:50.233726 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:50.233697 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9993cae3-8e92-479a-b182-0fe7be890338-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2df7b6-n2f5r\" (UID: \"9993cae3-8e92-479a-b182-0fe7be890338\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2df7b6-n2f5r" Oct 30 21:59:50.233875 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:50.233755 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9993cae3-8e92-479a-b182-0fe7be890338-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2df7b6-n2f5r\" (UID: \"9993cae3-8e92-479a-b182-0fe7be890338\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2df7b6-n2f5r" Oct 30 21:59:50.233875 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:50.233833 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5pts\" (UniqueName: \"kubernetes.io/projected/9993cae3-8e92-479a-b182-0fe7be890338-kube-api-access-q5pts\") pod \"maas-api-revoke-keys-e2e-api-b-2df7b6-n2f5r\" (UID: \"9993cae3-8e92-479a-b182-0fe7be890338\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2df7b6-n2f5r" Oct 30 21:59:50.335646 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:50.335574 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5pts\" (UniqueName: \"kubernetes.io/projected/9993cae3-8e92-479a-b182-0fe7be890338-kube-api-access-q5pts\") pod \"maas-api-revoke-keys-e2e-api-b-2df7b6-n2f5r\" (UID: \"9993cae3-8e92-479a-b182-0fe7be890338\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2df7b6-n2f5r" Oct 30 21:59:50.335810 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:50.335715 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9993cae3-8e92-479a-b182-0fe7be890338-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2df7b6-n2f5r\" (UID: \"9993cae3-8e92-479a-b182-0fe7be890338\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2df7b6-n2f5r" Oct 30 21:59:50.335810 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:50.335756 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9993cae3-8e92-479a-b182-0fe7be890338-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2df7b6-n2f5r\" (UID: \"9993cae3-8e92-479a-b182-0fe7be890338\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2df7b6-n2f5r" Oct 30 21:59:50.336452 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:50.336426 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9993cae3-8e92-479a-b182-0fe7be890338-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2df7b6-n2f5r\" (UID: \"9993cae3-8e92-479a-b182-0fe7be890338\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2df7b6-n2f5r" Oct 30 21:59:50.338133 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:50.338111 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9993cae3-8e92-479a-b182-0fe7be890338-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2df7b6-n2f5r\" (UID: \"9993cae3-8e92-479a-b182-0fe7be890338\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2df7b6-n2f5r" Oct 30 21:59:50.393421 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:50.393391 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5pts\" (UniqueName: \"kubernetes.io/projected/9993cae3-8e92-479a-b182-0fe7be890338-kube-api-access-q5pts\") pod \"maas-api-revoke-keys-e2e-api-b-2df7b6-n2f5r\" (UID: \"9993cae3-8e92-479a-b182-0fe7be890338\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2df7b6-n2f5r" Oct 30 21:59:50.675274 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:50.675196 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2df7b6-n2f5r" Oct 30 21:59:51.049976 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:51.049942 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2df7b6-n2f5r"] Oct 30 21:59:51.055664 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:59:51.055635 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9993cae3_8e92_479a_b182_0fe7be890338.slice/crio-58fc5572448d4179fd7e12856b2a4dd008ea0f8e33f9d6d8f202fe694b28280c WatchSource:0}: Error finding container 58fc5572448d4179fd7e12856b2a4dd008ea0f8e33f9d6d8f202fe694b28280c: Status 404 returned error can't find the container with id 58fc5572448d4179fd7e12856b2a4dd008ea0f8e33f9d6d8f202fe694b28280c Oct 30 21:59:51.350997 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:51.350972 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2df7b6-n2f5r" event={"ID":"9993cae3-8e92-479a-b182-0fe7be890338","Type":"ContainerStarted","Data":"a414703a772e27712f410033cbec2271cb1a5235da709c114d6b89c4ddd54e06"} Oct 30 21:59:51.351115 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:51.351020 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2df7b6-n2f5r" event={"ID":"9993cae3-8e92-479a-b182-0fe7be890338","Type":"ContainerStarted","Data":"58fc5572448d4179fd7e12856b2a4dd008ea0f8e33f9d6d8f202fe694b28280c"} Oct 30 21:59:51.407648 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:51.407619 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-40bb78-6qpv8"] Oct 30 21:59:51.411545 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:51.411528 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-40bb78-6qpv8" Oct 30 21:59:51.465620 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:51.465571 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2df7b6-n2f5r" podStartSLOduration=1.465557733 podStartE2EDuration="1.465557733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:51.463802515 +0000 UTC m=+2767.018502293" watchObservedRunningTime="2026-10-30 21:59:51.465557733 +0000 UTC m=+2767.020257536" Oct 30 21:59:51.477848 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:51.477821 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-40bb78-6qpv8"] Oct 30 21:59:51.549329 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:51.549290 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tpcz\" (UniqueName: \"kubernetes.io/projected/4632ebfb-a5e6-4c7f-9407-8dd343358cf2-kube-api-access-6tpcz\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-40bb78-6qpv8\" (UID: \"4632ebfb-a5e6-4c7f-9407-8dd343358cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-40bb78-6qpv8" Oct 30 21:59:51.549527 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:51.549507 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4632ebfb-a5e6-4c7f-9407-8dd343358cf2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-40bb78-6qpv8\" (UID: \"4632ebfb-a5e6-4c7f-9407-8dd343358cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-40bb78-6qpv8" Oct 30 21:59:51.549592 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:51.549578 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4632ebfb-a5e6-4c7f-9407-8dd343358cf2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-40bb78-6qpv8\" (UID: \"4632ebfb-a5e6-4c7f-9407-8dd343358cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-40bb78-6qpv8" Oct 30 21:59:51.650929 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:51.650823 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4632ebfb-a5e6-4c7f-9407-8dd343358cf2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-40bb78-6qpv8\" (UID: \"4632ebfb-a5e6-4c7f-9407-8dd343358cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-40bb78-6qpv8" Oct 30 21:59:51.651127 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:51.650959 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4632ebfb-a5e6-4c7f-9407-8dd343358cf2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-40bb78-6qpv8\" (UID: \"4632ebfb-a5e6-4c7f-9407-8dd343358cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-40bb78-6qpv8" Oct 30 21:59:51.651127 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:51.651057 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tpcz\" (UniqueName: \"kubernetes.io/projected/4632ebfb-a5e6-4c7f-9407-8dd343358cf2-kube-api-access-6tpcz\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-40bb78-6qpv8\" (UID: \"4632ebfb-a5e6-4c7f-9407-8dd343358cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-40bb78-6qpv8" Oct 30 21:59:51.651660 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:51.651636 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4632ebfb-a5e6-4c7f-9407-8dd343358cf2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-40bb78-6qpv8\" (UID: \"4632ebfb-a5e6-4c7f-9407-8dd343358cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-40bb78-6qpv8" Oct 30 21:59:51.653188 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:51.653164 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4632ebfb-a5e6-4c7f-9407-8dd343358cf2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-40bb78-6qpv8\" (UID: \"4632ebfb-a5e6-4c7f-9407-8dd343358cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-40bb78-6qpv8" Oct 30 21:59:51.709694 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:51.709654 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tpcz\" (UniqueName: \"kubernetes.io/projected/4632ebfb-a5e6-4c7f-9407-8dd343358cf2-kube-api-access-6tpcz\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-40bb78-6qpv8\" (UID: \"4632ebfb-a5e6-4c7f-9407-8dd343358cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-40bb78-6qpv8" Oct 30 21:59:51.724364 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:51.724325 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-40bb78-6qpv8" Oct 30 21:59:52.097055 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:52.097016 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-40bb78-6qpv8"] Oct 30 21:59:52.102298 ip-10-0-129-83 kubenswrapper[4560]: W1030 21:59:52.102271 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4632ebfb_a5e6_4c7f_9407_8dd343358cf2.slice/crio-c645c27f3c8c880aa8f108da8725fdb5a71d0e3c713519820b9ad09f508037f5 WatchSource:0}: Error finding container c645c27f3c8c880aa8f108da8725fdb5a71d0e3c713519820b9ad09f508037f5: Status 404 returned error can't find the container with id c645c27f3c8c880aa8f108da8725fdb5a71d0e3c713519820b9ad09f508037f5 Oct 30 21:59:52.358235 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:52.358206 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-40bb78-6qpv8" event={"ID":"4632ebfb-a5e6-4c7f-9407-8dd343358cf2","Type":"ContainerStarted","Data":"c645c27f3c8c880aa8f108da8725fdb5a71d0e3c713519820b9ad09f508037f5"} Oct 30 21:59:52.359831 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:52.359801 4560 generic.go:356] "Generic (PLEG): container finished" podID="9993cae3-8e92-479a-b182-0fe7be890338" containerID="a414703a772e27712f410033cbec2271cb1a5235da709c114d6b89c4ddd54e06" exitCode=0 Oct 30 21:59:52.359943 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:52.359844 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2df7b6-n2f5r" event={"ID":"9993cae3-8e92-479a-b182-0fe7be890338","Type":"ContainerDied","Data":"a414703a772e27712f410033cbec2271cb1a5235da709c114d6b89c4ddd54e06"} Oct 30 21:59:53.366962 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:53.366872 4560 generic.go:356] "Generic (PLEG): container finished" podID="4632ebfb-a5e6-4c7f-9407-8dd343358cf2" containerID="73de2528aad288f3ff8505acac417133efe07a9cd2829480460b4b7da5e29ff4" exitCode=0 Oct 30 21:59:53.366962 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:53.366907 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-40bb78-6qpv8" event={"ID":"4632ebfb-a5e6-4c7f-9407-8dd343358cf2","Type":"ContainerDied","Data":"73de2528aad288f3ff8505acac417133efe07a9cd2829480460b4b7da5e29ff4"} Oct 30 21:59:53.510001 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:53.509977 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2df7b6-n2f5r" Oct 30 21:59:53.676789 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:53.676719 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9993cae3-8e92-479a-b182-0fe7be890338-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9993cae3-8e92-479a-b182-0fe7be890338-maas-api-xks-ca\") pod \"9993cae3-8e92-479a-b182-0fe7be890338\" (UID: \"9993cae3-8e92-479a-b182-0fe7be890338\") " Oct 30 21:59:53.676789 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:53.676762 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9993cae3-8e92-479a-b182-0fe7be890338-kube-api-access-q5pts\" (UniqueName: \"kubernetes.io/projected/9993cae3-8e92-479a-b182-0fe7be890338-kube-api-access-q5pts\") pod \"9993cae3-8e92-479a-b182-0fe7be890338\" (UID: \"9993cae3-8e92-479a-b182-0fe7be890338\") " Oct 30 21:59:53.676980 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:53.676832 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9993cae3-8e92-479a-b182-0fe7be890338-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9993cae3-8e92-479a-b182-0fe7be890338-maas-api-openshift-ca\") pod \"9993cae3-8e92-479a-b182-0fe7be890338\" (UID: \"9993cae3-8e92-479a-b182-0fe7be890338\") " Oct 30 21:59:53.677243 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:53.677222 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9993cae3-8e92-479a-b182-0fe7be890338-maas-api-openshift-ca" pod "9993cae3-8e92-479a-b182-0fe7be890338" (UID: "9993cae3-8e92-479a-b182-0fe7be890338"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:53.678822 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:53.678784 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9993cae3-8e92-479a-b182-0fe7be890338-maas-api-xks-ca" pod "9993cae3-8e92-479a-b182-0fe7be890338" (UID: "9993cae3-8e92-479a-b182-0fe7be890338"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:53.678928 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:53.678878 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9993cae3-8e92-479a-b182-0fe7be890338-kube-api-access-q5pts" pod "9993cae3-8e92-479a-b182-0fe7be890338" (UID: "9993cae3-8e92-479a-b182-0fe7be890338"). InnerVolumeSpecName "kube-api-access-q5pts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:53.777964 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:53.777935 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9993cae3-8e92-479a-b182-0fe7be890338-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:53.777964 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:53.777961 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9993cae3-8e92-479a-b182-0fe7be890338-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:53.778131 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:53.777972 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5pts\" (UniqueName: \"kubernetes.io/projected/9993cae3-8e92-479a-b182-0fe7be890338-kube-api-access-q5pts\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:54.374223 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:54.374196 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2df7b6-n2f5r" Oct 30 21:59:54.374740 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:54.374231 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2df7b6-n2f5r" event={"ID":"9993cae3-8e92-479a-b182-0fe7be890338","Type":"ContainerDied","Data":"58fc5572448d4179fd7e12856b2a4dd008ea0f8e33f9d6d8f202fe694b28280c"} Oct 30 21:59:54.374740 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:54.374282 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58fc5572448d4179fd7e12856b2a4dd008ea0f8e33f9d6d8f202fe694b28280c" Oct 30 21:59:54.520141 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:54.520110 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-40bb78-6qpv8" Oct 30 21:59:54.688571 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:54.688491 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4632ebfb-a5e6-4c7f-9407-8dd343358cf2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4632ebfb-a5e6-4c7f-9407-8dd343358cf2-maas-api-xks-ca\") pod \"4632ebfb-a5e6-4c7f-9407-8dd343358cf2\" (UID: \"4632ebfb-a5e6-4c7f-9407-8dd343358cf2\") " Oct 30 21:59:54.688571 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:54.688537 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4632ebfb-a5e6-4c7f-9407-8dd343358cf2-kube-api-access-6tpcz\" (UniqueName: \"kubernetes.io/projected/4632ebfb-a5e6-4c7f-9407-8dd343358cf2-kube-api-access-6tpcz\") pod \"4632ebfb-a5e6-4c7f-9407-8dd343358cf2\" (UID: \"4632ebfb-a5e6-4c7f-9407-8dd343358cf2\") " Oct 30 21:59:54.688802 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:54.688701 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4632ebfb-a5e6-4c7f-9407-8dd343358cf2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4632ebfb-a5e6-4c7f-9407-8dd343358cf2-maas-api-openshift-ca\") pod \"4632ebfb-a5e6-4c7f-9407-8dd343358cf2\" (UID: \"4632ebfb-a5e6-4c7f-9407-8dd343358cf2\") " Oct 30 21:59:54.689451 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:54.689420 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4632ebfb-a5e6-4c7f-9407-8dd343358cf2-maas-api-openshift-ca" pod "4632ebfb-a5e6-4c7f-9407-8dd343358cf2" (UID: "4632ebfb-a5e6-4c7f-9407-8dd343358cf2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:54.690848 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:54.690823 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4632ebfb-a5e6-4c7f-9407-8dd343358cf2-maas-api-xks-ca" pod "4632ebfb-a5e6-4c7f-9407-8dd343358cf2" (UID: "4632ebfb-a5e6-4c7f-9407-8dd343358cf2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:54.691211 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:54.691185 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4632ebfb-a5e6-4c7f-9407-8dd343358cf2-kube-api-access-6tpcz" pod "4632ebfb-a5e6-4c7f-9407-8dd343358cf2" (UID: "4632ebfb-a5e6-4c7f-9407-8dd343358cf2"). InnerVolumeSpecName "kube-api-access-6tpcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:54.790130 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:54.790098 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4632ebfb-a5e6-4c7f-9407-8dd343358cf2-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:54.790130 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:54.790124 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4632ebfb-a5e6-4c7f-9407-8dd343358cf2-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:54.790130 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:54.790135 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tpcz\" (UniqueName: \"kubernetes.io/projected/4632ebfb-a5e6-4c7f-9407-8dd343358cf2-kube-api-access-6tpcz\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:55.381565 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:55.381537 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-40bb78-6qpv8" Oct 30 21:59:55.381998 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:55.381580 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-40bb78-6qpv8" event={"ID":"4632ebfb-a5e6-4c7f-9407-8dd343358cf2","Type":"ContainerDied","Data":"c645c27f3c8c880aa8f108da8725fdb5a71d0e3c713519820b9ad09f508037f5"} Oct 30 21:59:55.381998 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:55.381639 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c645c27f3c8c880aa8f108da8725fdb5a71d0e3c713519820b9ad09f508037f5" Oct 30 21:59:57.905455 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:57.905421 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-167561-dqlmp"] Oct 30 21:59:57.951986 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:57.951947 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-167561-dqlmp"] Oct 30 21:59:58.576084 ip-10-0-129-83 kubenswrapper[4560]: I1030 21:59:58.576046 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b831442-fe0b-42ea-8ea2-16313d31dd46" path="/var/lib/kubelet/pods/3b831442-fe0b-42ea-8ea2-16313d31dd46/volumes" Oct 30 22:00:00.179442 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.179405 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bfdf30b9-29889960-4c9wb"] Oct 30 22:00:00.180036 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.180013 4560 cpu_manager.go:414] "containerMap: removing container" podUID="4632ebfb-a5e6-4c7f-9407-8dd343358cf2" containerName="revoke-keys" Oct 30 22:00:00.180115 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.180038 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4632ebfb-a5e6-4c7f-9407-8dd343358cf2" containerName="revoke-keys" Oct 30 22:00:00.180115 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.180066 4560 cpu_manager.go:414] "containerMap: removing container" podUID="9993cae3-8e92-479a-b182-0fe7be890338" containerName="revoke-keys" Oct 30 22:00:00.180115 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.180073 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9993cae3-8e92-479a-b182-0fe7be890338" containerName="revoke-keys" Oct 30 22:00:00.180279 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.180222 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="9993cae3-8e92-479a-b182-0fe7be890338" containerName="revoke-keys" Oct 30 22:00:00.180279 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.180246 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="4632ebfb-a5e6-4c7f-9407-8dd343358cf2" containerName="revoke-keys" Oct 30 22:00:00.184721 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.184694 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bfdf30b9-29889960-4c9wb" Oct 30 22:00:00.186905 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.186884 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-40bb78-29889960-98vl4"] Oct 30 22:00:00.190195 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.190179 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-40bb78-29889960-98vl4" Oct 30 22:00:00.203755 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.203732 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889960-g9zpz"] Oct 30 22:00:00.208555 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.208427 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889960-g9zpz" Oct 30 22:00:00.226321 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.226299 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19dd4f-29889960-hcm42"] Oct 30 22:00:00.229746 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.229730 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889960-tfsx2"] Oct 30 22:00:00.229847 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.229828 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19dd4f-29889960-hcm42" Oct 30 22:00:00.233310 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.233296 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889960-tfsx2" Oct 30 22:00:00.238801 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.238780 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889960-vqpcn"] Oct 30 22:00:00.242130 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.242110 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889960-vqpcn" Oct 30 22:00:00.246349 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.246304 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:00:00.246349 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.246320 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:00:00.246483 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.246393 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:00:00.273740 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.273069 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6ll8\" (UniqueName: \"kubernetes.io/projected/a24ad108-152e-4690-82ae-71e98f901c61-kube-api-access-n6ll8\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-40bb78-29889960-98vl4\" (UID: \"a24ad108-152e-4690-82ae-71e98f901c61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-40bb78-29889960-98vl4" Oct 30 22:00:00.273740 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.273118 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wbh2\" (UniqueName: \"kubernetes.io/projected/e174ecd3-9e50-458b-85aa-b130ceba6d70-kube-api-access-2wbh2\") pod \"maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889960-vqpcn\" (UID: \"e174ecd3-9e50-458b-85aa-b130ceba6d70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889960-vqpcn" Oct 30 22:00:00.273740 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.273144 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdd9p\" (UniqueName: \"kubernetes.io/projected/45e6d264-3503-493e-95f2-6308e04e93ab-kube-api-access-hdd9p\") pod \"maas-api-key-cleanup-e2e-worker-w3-71aab5-29889960-g9zpz\" (UID: \"45e6d264-3503-493e-95f2-6308e04e93ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889960-g9zpz" Oct 30 22:00:00.273740 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.273205 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22606928-4bc3-49e5-91e9-00497b022b10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-bfdf30b9-29889960-4c9wb\" (UID: \"22606928-4bc3-49e5-91e9-00497b022b10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bfdf30b9-29889960-4c9wb" Oct 30 22:00:00.273740 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.273241 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptrhq\" (UniqueName: \"kubernetes.io/projected/818309d7-73a4-404f-9942-5bcbb2a4fd14-kube-api-access-ptrhq\") pod \"maas-api-key-cleanup-e2e-worker-w4-554f46-29889960-tfsx2\" (UID: \"818309d7-73a4-404f-9942-5bcbb2a4fd14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889960-tfsx2" Oct 30 22:00:00.273740 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.273282 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/818309d7-73a4-404f-9942-5bcbb2a4fd14-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-554f46-29889960-tfsx2\" (UID: \"818309d7-73a4-404f-9942-5bcbb2a4fd14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889960-tfsx2" Oct 30 22:00:00.273740 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.273316 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx2fc\" (UniqueName: \"kubernetes.io/projected/aa5fbac8-0bd8-4793-bbaa-c8dae197566d-kube-api-access-nx2fc\") pod \"maas-api-key-cleanup-e2e-worker-w1-19dd4f-29889960-hcm42\" (UID: \"aa5fbac8-0bd8-4793-bbaa-c8dae197566d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19dd4f-29889960-hcm42" Oct 30 22:00:00.273740 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.273423 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e174ecd3-9e50-458b-85aa-b130ceba6d70-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889960-vqpcn\" (UID: \"e174ecd3-9e50-458b-85aa-b130ceba6d70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889960-vqpcn" Oct 30 22:00:00.273740 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.273447 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45e6d264-3503-493e-95f2-6308e04e93ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-71aab5-29889960-g9zpz\" (UID: \"45e6d264-3503-493e-95f2-6308e04e93ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889960-g9zpz" Oct 30 22:00:00.273740 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.273524 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa5fbac8-0bd8-4793-bbaa-c8dae197566d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-19dd4f-29889960-hcm42\" (UID: \"aa5fbac8-0bd8-4793-bbaa-c8dae197566d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19dd4f-29889960-hcm42" Oct 30 22:00:00.273740 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.273569 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wppj5\" (UniqueName: \"kubernetes.io/projected/22606928-4bc3-49e5-91e9-00497b022b10-kube-api-access-wppj5\") pod \"maas-api-key-cleanup-e2e-mt-bfdf30b9-29889960-4c9wb\" (UID: \"22606928-4bc3-49e5-91e9-00497b022b10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bfdf30b9-29889960-4c9wb" Oct 30 22:00:00.273740 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.273607 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a24ad108-152e-4690-82ae-71e98f901c61-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-40bb78-29889960-98vl4\" (UID: \"a24ad108-152e-4690-82ae-71e98f901c61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-40bb78-29889960-98vl4" Oct 30 22:00:00.275169 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.275149 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-bzl5x"] Oct 30 22:00:00.278733 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.278712 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bfdf30b9-29889960-4c9wb"] Oct 30 22:00:00.278818 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.278799 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-bzl5x" Oct 30 22:00:00.285947 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.285926 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ca0a55-29889960-8wsx6"] Oct 30 22:00:00.290014 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.289996 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ca0a55-29889960-8wsx6" Oct 30 22:00:00.299241 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.299218 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-40bb78-29889960-98vl4"] Oct 30 22:00:00.304406 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.303291 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889960-g9zpz"] Oct 30 22:00:00.306700 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.306681 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2df7b6-29889960-96qhk"] Oct 30 22:00:00.320518 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.320494 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19dd4f-29889960-hcm42"] Oct 30 22:00:00.320661 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.320645 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2df7b6-29889960-96qhk" Oct 30 22:00:00.320897 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.320873 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889960-tfsx2"] Oct 30 22:00:00.326148 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.326125 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ed6135-29889960-bfnmd"] Oct 30 22:00:00.330188 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.330171 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ed6135-29889960-bfnmd" Oct 30 22:00:00.370834 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.370797 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ca0a55-29889960-8wsx6"] Oct 30 22:00:00.374836 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.374809 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/818309d7-73a4-404f-9942-5bcbb2a4fd14-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-554f46-29889960-tfsx2\" (UID: \"818309d7-73a4-404f-9942-5bcbb2a4fd14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889960-tfsx2" Oct 30 22:00:00.374954 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.374863 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx2fc\" (UniqueName: \"kubernetes.io/projected/aa5fbac8-0bd8-4793-bbaa-c8dae197566d-kube-api-access-nx2fc\") pod \"maas-api-key-cleanup-e2e-worker-w1-19dd4f-29889960-hcm42\" (UID: \"aa5fbac8-0bd8-4793-bbaa-c8dae197566d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19dd4f-29889960-hcm42" Oct 30 22:00:00.374954 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.374914 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfn98\" (UniqueName: \"kubernetes.io/projected/ae04a1f9-8997-42e2-b7e9-94e84fd97d71-kube-api-access-bfn98\") pod \"maas-api-key-cleanup-29889960-bzl5x\" (UID: \"ae04a1f9-8997-42e2-b7e9-94e84fd97d71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-bzl5x" Oct 30 22:00:00.374954 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.374948 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08796630-f181-4b69-bd3c-32f627dff3ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-2df7b6-29889960-96qhk\" (UID: \"08796630-f181-4b69-bd3c-32f627dff3ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2df7b6-29889960-96qhk" Oct 30 22:00:00.375460 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.374998 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e174ecd3-9e50-458b-85aa-b130ceba6d70-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889960-vqpcn\" (UID: \"e174ecd3-9e50-458b-85aa-b130ceba6d70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889960-vqpcn" Oct 30 22:00:00.375460 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.375025 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45e6d264-3503-493e-95f2-6308e04e93ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-71aab5-29889960-g9zpz\" (UID: \"45e6d264-3503-493e-95f2-6308e04e93ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889960-g9zpz" Oct 30 22:00:00.375460 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.375160 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d29a3644-c4e8-40eb-a34b-f531c2a1565b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ed6135-29889960-bfnmd\" (UID: \"d29a3644-c4e8-40eb-a34b-f531c2a1565b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ed6135-29889960-bfnmd" Oct 30 22:00:00.375460 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.375202 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptw5z\" (UniqueName: \"kubernetes.io/projected/08796630-f181-4b69-bd3c-32f627dff3ac-kube-api-access-ptw5z\") pod \"maas-api-key-cleanup-e2e-api-b-2df7b6-29889960-96qhk\" (UID: \"08796630-f181-4b69-bd3c-32f627dff3ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2df7b6-29889960-96qhk" Oct 30 22:00:00.375460 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.375272 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa5fbac8-0bd8-4793-bbaa-c8dae197566d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-19dd4f-29889960-hcm42\" (UID: \"aa5fbac8-0bd8-4793-bbaa-c8dae197566d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19dd4f-29889960-hcm42" Oct 30 22:00:00.375460 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.375354 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wppj5\" (UniqueName: \"kubernetes.io/projected/22606928-4bc3-49e5-91e9-00497b022b10-kube-api-access-wppj5\") pod \"maas-api-key-cleanup-e2e-mt-bfdf30b9-29889960-4c9wb\" (UID: \"22606928-4bc3-49e5-91e9-00497b022b10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bfdf30b9-29889960-4c9wb" Oct 30 22:00:00.375460 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.375396 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a24ad108-152e-4690-82ae-71e98f901c61-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-40bb78-29889960-98vl4\" (UID: \"a24ad108-152e-4690-82ae-71e98f901c61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-40bb78-29889960-98vl4" Oct 30 22:00:00.375460 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.375436 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6h7s\" (UniqueName: \"kubernetes.io/projected/d29a3644-c4e8-40eb-a34b-f531c2a1565b-kube-api-access-n6h7s\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ed6135-29889960-bfnmd\" (UID: \"d29a3644-c4e8-40eb-a34b-f531c2a1565b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ed6135-29889960-bfnmd" Oct 30 22:00:00.377998 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.375472 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2z4c\" (UniqueName: \"kubernetes.io/projected/d8a90e7a-b379-4742-a167-01aabd410bf9-kube-api-access-g2z4c\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-ca0a55-29889960-8wsx6\" (UID: \"d8a90e7a-b379-4742-a167-01aabd410bf9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ca0a55-29889960-8wsx6" Oct 30 22:00:00.377998 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.375506 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae04a1f9-8997-42e2-b7e9-94e84fd97d71-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889960-bzl5x\" (UID: \"ae04a1f9-8997-42e2-b7e9-94e84fd97d71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-bzl5x" Oct 30 22:00:00.377998 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.375541 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6ll8\" (UniqueName: \"kubernetes.io/projected/a24ad108-152e-4690-82ae-71e98f901c61-kube-api-access-n6ll8\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-40bb78-29889960-98vl4\" (UID: \"a24ad108-152e-4690-82ae-71e98f901c61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-40bb78-29889960-98vl4" Oct 30 22:00:00.377998 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.375579 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wbh2\" (UniqueName: \"kubernetes.io/projected/e174ecd3-9e50-458b-85aa-b130ceba6d70-kube-api-access-2wbh2\") pod \"maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889960-vqpcn\" (UID: \"e174ecd3-9e50-458b-85aa-b130ceba6d70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889960-vqpcn" Oct 30 22:00:00.377998 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.375604 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdd9p\" (UniqueName: \"kubernetes.io/projected/45e6d264-3503-493e-95f2-6308e04e93ab-kube-api-access-hdd9p\") pod \"maas-api-key-cleanup-e2e-worker-w3-71aab5-29889960-g9zpz\" (UID: \"45e6d264-3503-493e-95f2-6308e04e93ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889960-g9zpz" Oct 30 22:00:00.377998 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.375678 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a90e7a-b379-4742-a167-01aabd410bf9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-ca0a55-29889960-8wsx6\" (UID: \"d8a90e7a-b379-4742-a167-01aabd410bf9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ca0a55-29889960-8wsx6" Oct 30 22:00:00.377998 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.375718 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22606928-4bc3-49e5-91e9-00497b022b10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-bfdf30b9-29889960-4c9wb\" (UID: \"22606928-4bc3-49e5-91e9-00497b022b10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bfdf30b9-29889960-4c9wb" Oct 30 22:00:00.377998 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.375763 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptrhq\" (UniqueName: \"kubernetes.io/projected/818309d7-73a4-404f-9942-5bcbb2a4fd14-kube-api-access-ptrhq\") pod \"maas-api-key-cleanup-e2e-worker-w4-554f46-29889960-tfsx2\" (UID: \"818309d7-73a4-404f-9942-5bcbb2a4fd14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889960-tfsx2" Oct 30 22:00:00.377998 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.375804 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e174ecd3-9e50-458b-85aa-b130ceba6d70-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889960-vqpcn\" (UID: \"e174ecd3-9e50-458b-85aa-b130ceba6d70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889960-vqpcn" Oct 30 22:00:00.377998 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.375913 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45e6d264-3503-493e-95f2-6308e04e93ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-71aab5-29889960-g9zpz\" (UID: \"45e6d264-3503-493e-95f2-6308e04e93ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889960-g9zpz" Oct 30 22:00:00.377998 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.375996 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/818309d7-73a4-404f-9942-5bcbb2a4fd14-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-554f46-29889960-tfsx2\" (UID: \"818309d7-73a4-404f-9942-5bcbb2a4fd14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889960-tfsx2" Oct 30 22:00:00.377998 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.376793 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a24ad108-152e-4690-82ae-71e98f901c61-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-40bb78-29889960-98vl4\" (UID: \"a24ad108-152e-4690-82ae-71e98f901c61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-40bb78-29889960-98vl4" Oct 30 22:00:00.377998 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.376798 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa5fbac8-0bd8-4793-bbaa-c8dae197566d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-19dd4f-29889960-hcm42\" (UID: \"aa5fbac8-0bd8-4793-bbaa-c8dae197566d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19dd4f-29889960-hcm42" Oct 30 22:00:00.377998 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.377449 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22606928-4bc3-49e5-91e9-00497b022b10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-bfdf30b9-29889960-4c9wb\" (UID: \"22606928-4bc3-49e5-91e9-00497b022b10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bfdf30b9-29889960-4c9wb" Oct 30 22:00:00.381980 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.381959 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889960-vqpcn"] Oct 30 22:00:00.388250 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.388230 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2df7b6-29889960-96qhk"] Oct 30 22:00:00.391698 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.391677 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-bzl5x"] Oct 30 22:00:00.403374 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.403330 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ed6135-29889960-bfnmd"] Oct 30 22:00:00.443172 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.443096 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wppj5\" (UniqueName: \"kubernetes.io/projected/22606928-4bc3-49e5-91e9-00497b022b10-kube-api-access-wppj5\") pod \"maas-api-key-cleanup-e2e-mt-bfdf30b9-29889960-4c9wb\" (UID: \"22606928-4bc3-49e5-91e9-00497b022b10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bfdf30b9-29889960-4c9wb" Oct 30 22:00:00.443172 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.443139 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wbh2\" (UniqueName: \"kubernetes.io/projected/e174ecd3-9e50-458b-85aa-b130ceba6d70-kube-api-access-2wbh2\") pod \"maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889960-vqpcn\" (UID: \"e174ecd3-9e50-458b-85aa-b130ceba6d70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889960-vqpcn" Oct 30 22:00:00.443172 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.443106 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptrhq\" (UniqueName: \"kubernetes.io/projected/818309d7-73a4-404f-9942-5bcbb2a4fd14-kube-api-access-ptrhq\") pod \"maas-api-key-cleanup-e2e-worker-w4-554f46-29889960-tfsx2\" (UID: \"818309d7-73a4-404f-9942-5bcbb2a4fd14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889960-tfsx2" Oct 30 22:00:00.444369 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.444326 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdd9p\" (UniqueName: \"kubernetes.io/projected/45e6d264-3503-493e-95f2-6308e04e93ab-kube-api-access-hdd9p\") pod \"maas-api-key-cleanup-e2e-worker-w3-71aab5-29889960-g9zpz\" (UID: \"45e6d264-3503-493e-95f2-6308e04e93ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889960-g9zpz" Oct 30 22:00:00.444623 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.444607 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx2fc\" (UniqueName: \"kubernetes.io/projected/aa5fbac8-0bd8-4793-bbaa-c8dae197566d-kube-api-access-nx2fc\") pod \"maas-api-key-cleanup-e2e-worker-w1-19dd4f-29889960-hcm42\" (UID: \"aa5fbac8-0bd8-4793-bbaa-c8dae197566d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19dd4f-29889960-hcm42" Oct 30 22:00:00.455252 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.455235 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6ll8\" (UniqueName: \"kubernetes.io/projected/a24ad108-152e-4690-82ae-71e98f901c61-kube-api-access-n6ll8\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-40bb78-29889960-98vl4\" (UID: \"a24ad108-152e-4690-82ae-71e98f901c61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-40bb78-29889960-98vl4" Oct 30 22:00:00.477070 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.477049 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfn98\" (UniqueName: \"kubernetes.io/projected/ae04a1f9-8997-42e2-b7e9-94e84fd97d71-kube-api-access-bfn98\") pod \"maas-api-key-cleanup-29889960-bzl5x\" (UID: \"ae04a1f9-8997-42e2-b7e9-94e84fd97d71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-bzl5x" Oct 30 22:00:00.477169 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.477092 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08796630-f181-4b69-bd3c-32f627dff3ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-2df7b6-29889960-96qhk\" (UID: \"08796630-f181-4b69-bd3c-32f627dff3ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2df7b6-29889960-96qhk" Oct 30 22:00:00.477240 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.477171 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d29a3644-c4e8-40eb-a34b-f531c2a1565b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ed6135-29889960-bfnmd\" (UID: \"d29a3644-c4e8-40eb-a34b-f531c2a1565b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ed6135-29889960-bfnmd" Oct 30 22:00:00.477240 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.477198 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptw5z\" (UniqueName: \"kubernetes.io/projected/08796630-f181-4b69-bd3c-32f627dff3ac-kube-api-access-ptw5z\") pod \"maas-api-key-cleanup-e2e-api-b-2df7b6-29889960-96qhk\" (UID: \"08796630-f181-4b69-bd3c-32f627dff3ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2df7b6-29889960-96qhk" Oct 30 22:00:00.477388 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.477282 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6h7s\" (UniqueName: \"kubernetes.io/projected/d29a3644-c4e8-40eb-a34b-f531c2a1565b-kube-api-access-n6h7s\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ed6135-29889960-bfnmd\" (UID: \"d29a3644-c4e8-40eb-a34b-f531c2a1565b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ed6135-29889960-bfnmd" Oct 30 22:00:00.477388 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.477316 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2z4c\" (UniqueName: \"kubernetes.io/projected/d8a90e7a-b379-4742-a167-01aabd410bf9-kube-api-access-g2z4c\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-ca0a55-29889960-8wsx6\" (UID: \"d8a90e7a-b379-4742-a167-01aabd410bf9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ca0a55-29889960-8wsx6" Oct 30 22:00:00.477388 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.477366 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae04a1f9-8997-42e2-b7e9-94e84fd97d71-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889960-bzl5x\" (UID: \"ae04a1f9-8997-42e2-b7e9-94e84fd97d71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-bzl5x" Oct 30 22:00:00.477542 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.477445 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a90e7a-b379-4742-a167-01aabd410bf9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-ca0a55-29889960-8wsx6\" (UID: \"d8a90e7a-b379-4742-a167-01aabd410bf9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ca0a55-29889960-8wsx6" Oct 30 22:00:00.477868 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.477846 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08796630-f181-4b69-bd3c-32f627dff3ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-2df7b6-29889960-96qhk\" (UID: \"08796630-f181-4b69-bd3c-32f627dff3ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2df7b6-29889960-96qhk" Oct 30 22:00:00.477937 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.477915 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d29a3644-c4e8-40eb-a34b-f531c2a1565b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ed6135-29889960-bfnmd\" (UID: \"d29a3644-c4e8-40eb-a34b-f531c2a1565b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ed6135-29889960-bfnmd" Oct 30 22:00:00.478083 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.478066 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae04a1f9-8997-42e2-b7e9-94e84fd97d71-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889960-bzl5x\" (UID: \"ae04a1f9-8997-42e2-b7e9-94e84fd97d71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-bzl5x" Oct 30 22:00:00.478139 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.478119 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a90e7a-b379-4742-a167-01aabd410bf9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-ca0a55-29889960-8wsx6\" (UID: \"d8a90e7a-b379-4742-a167-01aabd410bf9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ca0a55-29889960-8wsx6" Oct 30 22:00:00.497931 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.497913 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bfdf30b9-29889960-4c9wb" Oct 30 22:00:00.518865 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.518848 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-40bb78-29889960-98vl4" Oct 30 22:00:00.526906 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.526886 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889960-g9zpz" Oct 30 22:00:00.543932 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.543902 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19dd4f-29889960-hcm42" Oct 30 22:00:00.550980 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.550844 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889960-tfsx2" Oct 30 22:00:00.557007 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.556834 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6h7s\" (UniqueName: \"kubernetes.io/projected/d29a3644-c4e8-40eb-a34b-f531c2a1565b-kube-api-access-n6h7s\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ed6135-29889960-bfnmd\" (UID: \"d29a3644-c4e8-40eb-a34b-f531c2a1565b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ed6135-29889960-bfnmd" Oct 30 22:00:00.557206 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.557059 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889960-vqpcn" Oct 30 22:00:00.558388 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.558285 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfn98\" (UniqueName: \"kubernetes.io/projected/ae04a1f9-8997-42e2-b7e9-94e84fd97d71-kube-api-access-bfn98\") pod \"maas-api-key-cleanup-29889960-bzl5x\" (UID: \"ae04a1f9-8997-42e2-b7e9-94e84fd97d71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-bzl5x" Oct 30 22:00:00.560881 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.560855 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptw5z\" (UniqueName: \"kubernetes.io/projected/08796630-f181-4b69-bd3c-32f627dff3ac-kube-api-access-ptw5z\") pod \"maas-api-key-cleanup-e2e-api-b-2df7b6-29889960-96qhk\" (UID: \"08796630-f181-4b69-bd3c-32f627dff3ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2df7b6-29889960-96qhk" Oct 30 22:00:00.568050 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.567110 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2z4c\" (UniqueName: \"kubernetes.io/projected/d8a90e7a-b379-4742-a167-01aabd410bf9-kube-api-access-g2z4c\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-ca0a55-29889960-8wsx6\" (UID: \"d8a90e7a-b379-4742-a167-01aabd410bf9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ca0a55-29889960-8wsx6" Oct 30 22:00:00.606030 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.604128 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-bzl5x" Oct 30 22:00:00.621544 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.621516 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ca0a55-29889960-8wsx6" Oct 30 22:00:00.642653 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.642590 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2df7b6-29889960-96qhk" Oct 30 22:00:00.649387 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.649192 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ed6135-29889960-bfnmd" Oct 30 22:00:00.900227 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.900193 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bfdf30b9-29889960-4c9wb"] Oct 30 22:00:00.904093 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:00:00.904068 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22606928_4bc3_49e5_91e9_00497b022b10.slice/crio-8bcb156bbbd327b814310f20b73893ebde42ece43c31cc0f7a7886ab36a857a6 WatchSource:0}: Error finding container 8bcb156bbbd327b814310f20b73893ebde42ece43c31cc0f7a7886ab36a857a6: Status 404 returned error can't find the container with id 8bcb156bbbd327b814310f20b73893ebde42ece43c31cc0f7a7886ab36a857a6 Oct 30 22:00:00.948569 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.948529 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-40bb78-29889960-98vl4"] Oct 30 22:00:00.953836 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:00:00.953768 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda24ad108_152e_4690_82ae_71e98f901c61.slice/crio-4431c088644cbb2d50d13ad1a1d22156c486ddfcd4cf74196adddeacdc74e2aa WatchSource:0}: Error finding container 4431c088644cbb2d50d13ad1a1d22156c486ddfcd4cf74196adddeacdc74e2aa: Status 404 returned error can't find the container with id 4431c088644cbb2d50d13ad1a1d22156c486ddfcd4cf74196adddeacdc74e2aa Oct 30 22:00:00.980490 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:00.980468 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889960-g9zpz"] Oct 30 22:00:00.997458 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:00:00.997435 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e6d264_3503_493e_95f2_6308e04e93ab.slice/crio-f5a024db04d55021e6d3dd201b5932cefc01462dc3c84f7fb8b121dd15f68b99 WatchSource:0}: Error finding container f5a024db04d55021e6d3dd201b5932cefc01462dc3c84f7fb8b121dd15f68b99: Status 404 returned error can't find the container with id f5a024db04d55021e6d3dd201b5932cefc01462dc3c84f7fb8b121dd15f68b99 Oct 30 22:00:01.021272 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:01.021249 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19dd4f-29889960-hcm42"] Oct 30 22:00:01.031816 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:01.031792 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889960-vqpcn"] Oct 30 22:00:01.076291 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:01.076265 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889960-tfsx2"] Oct 30 22:00:01.115222 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:01.115194 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-bzl5x"] Oct 30 22:00:01.123352 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:01.123322 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ca0a55-29889960-8wsx6"] Oct 30 22:00:01.152731 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:01.152511 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ed6135-29889960-bfnmd"] Oct 30 22:00:01.153111 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:00:01.153081 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8a90e7a_b379_4742_a167_01aabd410bf9.slice/crio-9f0a876bfd508670a4e144a241a2003a7cd77ef8283c6934c3263cb98edc57b7 WatchSource:0}: Error finding container 9f0a876bfd508670a4e144a241a2003a7cd77ef8283c6934c3263cb98edc57b7: Status 404 returned error can't find the container with id 9f0a876bfd508670a4e144a241a2003a7cd77ef8283c6934c3263cb98edc57b7 Oct 30 22:00:01.156920 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:01.156897 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2df7b6-29889960-96qhk"] Oct 30 22:00:01.168594 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:00:01.168497 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08796630_f181_4b69_bd3c_32f627dff3ac.slice/crio-567b7d39a2ae638f9da801bfd351d496cebe212be001f4b67936d05512e58b69 WatchSource:0}: Error finding container 567b7d39a2ae638f9da801bfd351d496cebe212be001f4b67936d05512e58b69: Status 404 returned error can't find the container with id 567b7d39a2ae638f9da801bfd351d496cebe212be001f4b67936d05512e58b69 Oct 30 22:00:01.418008 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:01.417973 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889960-tfsx2" event={"ID":"818309d7-73a4-404f-9942-5bcbb2a4fd14","Type":"ContainerStarted","Data":"f1b4a50a0245f1214ee07b38a5fd309501b73f644432a08bf8b639f8ac97ec79"} Oct 30 22:00:01.419295 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:01.419271 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19dd4f-29889960-hcm42" event={"ID":"aa5fbac8-0bd8-4793-bbaa-c8dae197566d","Type":"ContainerStarted","Data":"1508e941687c973b8b3b6f3df8d3c83c0e99799842abf263d6d41278e09c012b"} Oct 30 22:00:01.420493 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:01.420468 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-bzl5x" event={"ID":"ae04a1f9-8997-42e2-b7e9-94e84fd97d71","Type":"ContainerStarted","Data":"3b00a2f4eef5eef01c3cf727040ca9147ec1e6750a2441c6aea337f1aa0741d4"} Oct 30 22:00:01.421609 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:01.421589 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ca0a55-29889960-8wsx6" event={"ID":"d8a90e7a-b379-4742-a167-01aabd410bf9","Type":"ContainerStarted","Data":"9f0a876bfd508670a4e144a241a2003a7cd77ef8283c6934c3263cb98edc57b7"} Oct 30 22:00:01.422649 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:01.422626 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bfdf30b9-29889960-4c9wb" event={"ID":"22606928-4bc3-49e5-91e9-00497b022b10","Type":"ContainerStarted","Data":"8bcb156bbbd327b814310f20b73893ebde42ece43c31cc0f7a7886ab36a857a6"} Oct 30 22:00:01.423660 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:01.423641 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889960-g9zpz" event={"ID":"45e6d264-3503-493e-95f2-6308e04e93ab","Type":"ContainerStarted","Data":"f5a024db04d55021e6d3dd201b5932cefc01462dc3c84f7fb8b121dd15f68b99"} Oct 30 22:00:01.424640 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:01.424621 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2df7b6-29889960-96qhk" event={"ID":"08796630-f181-4b69-bd3c-32f627dff3ac","Type":"ContainerStarted","Data":"567b7d39a2ae638f9da801bfd351d496cebe212be001f4b67936d05512e58b69"} Oct 30 22:00:01.425683 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:01.425666 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ed6135-29889960-bfnmd" event={"ID":"d29a3644-c4e8-40eb-a34b-f531c2a1565b","Type":"ContainerStarted","Data":"8738dc6c6fca3c51faf1d9f91c9aaed08dc307540a3506aea34bcbffd86a97eb"} Oct 30 22:00:01.426923 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:01.426903 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889960-vqpcn" event={"ID":"e174ecd3-9e50-458b-85aa-b130ceba6d70","Type":"ContainerStarted","Data":"f2f202cca41bbb2b95701b97e27307a9d61d604683010afc06138fc38a0888e8"} Oct 30 22:00:01.428742 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:01.428720 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-40bb78-29889960-98vl4" event={"ID":"a24ad108-152e-4690-82ae-71e98f901c61","Type":"ContainerStarted","Data":"15263b3fdd4c8e0307a7b5b141ac324df2ce1c80cdca89ee758e4060d2cfc5fd"} Oct 30 22:00:01.428823 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:01.428756 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-40bb78-29889960-98vl4" event={"ID":"a24ad108-152e-4690-82ae-71e98f901c61","Type":"ContainerStarted","Data":"4431c088644cbb2d50d13ad1a1d22156c486ddfcd4cf74196adddeacdc74e2aa"} Oct 30 22:00:01.565672 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:01.565621 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-40bb78-29889960-98vl4" podStartSLOduration=1.565606785 podStartE2EDuration="1.565606785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:01.563989559 +0000 UTC m=+2777.118689337" watchObservedRunningTime="2026-10-30 22:00:01.565606785 +0000 UTC m=+2777.120306588" Oct 30 22:00:02.438450 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:02.438415 4560 generic.go:356] "Generic (PLEG): container finished" podID="45e6d264-3503-493e-95f2-6308e04e93ab" containerID="15ffa9ae5d631bbc0cbb9b7225ce4a22dc0e0d22691ee1046d53ee0d38731298" exitCode=60 Oct 30 22:00:02.438852 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:02.438497 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889960-g9zpz" event={"ID":"45e6d264-3503-493e-95f2-6308e04e93ab","Type":"ContainerDied","Data":"15ffa9ae5d631bbc0cbb9b7225ce4a22dc0e0d22691ee1046d53ee0d38731298"} Oct 30 22:00:02.438852 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:02.438784 4560 scope.go:122] "RemoveContainer" containerID="15ffa9ae5d631bbc0cbb9b7225ce4a22dc0e0d22691ee1046d53ee0d38731298" Oct 30 22:00:02.440328 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:02.440232 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2df7b6-29889960-96qhk" event={"ID":"08796630-f181-4b69-bd3c-32f627dff3ac","Type":"ContainerStarted","Data":"30152c6086bd8117f44d7b8f585d4d808f78da845273f76bc24213b6ea936c32"} Oct 30 22:00:02.440586 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:02.440566 4560 scope.go:122] "RemoveContainer" containerID="30152c6086bd8117f44d7b8f585d4d808f78da845273f76bc24213b6ea936c32" Oct 30 22:00:02.443657 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:02.441856 4560 generic.go:356] "Generic (PLEG): container finished" podID="d29a3644-c4e8-40eb-a34b-f531c2a1565b" containerID="c111b5384126890cef4f2c52680980aebc3cc4548777d0a2cc71c5b5e03f0e9a" exitCode=60 Oct 30 22:00:02.443657 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:02.441936 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ed6135-29889960-bfnmd" event={"ID":"d29a3644-c4e8-40eb-a34b-f531c2a1565b","Type":"ContainerDied","Data":"c111b5384126890cef4f2c52680980aebc3cc4548777d0a2cc71c5b5e03f0e9a"} Oct 30 22:00:02.443657 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:02.442120 4560 scope.go:122] "RemoveContainer" containerID="c111b5384126890cef4f2c52680980aebc3cc4548777d0a2cc71c5b5e03f0e9a" Oct 30 22:00:02.443937 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:02.443679 4560 generic.go:356] "Generic (PLEG): container finished" podID="e174ecd3-9e50-458b-85aa-b130ceba6d70" containerID="edb3d28e5734985e8d7b1dfa0ac4090cd33d2bcb8ff90d28568c7135a929339f" exitCode=60 Oct 30 22:00:02.443937 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:02.443753 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889960-vqpcn" event={"ID":"e174ecd3-9e50-458b-85aa-b130ceba6d70","Type":"ContainerDied","Data":"edb3d28e5734985e8d7b1dfa0ac4090cd33d2bcb8ff90d28568c7135a929339f"} Oct 30 22:00:02.443937 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:02.443931 4560 scope.go:122] "RemoveContainer" containerID="edb3d28e5734985e8d7b1dfa0ac4090cd33d2bcb8ff90d28568c7135a929339f" Oct 30 22:00:02.445349 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:02.445317 4560 generic.go:356] "Generic (PLEG): container finished" podID="a24ad108-152e-4690-82ae-71e98f901c61" containerID="15263b3fdd4c8e0307a7b5b141ac324df2ce1c80cdca89ee758e4060d2cfc5fd" exitCode=60 Oct 30 22:00:02.445436 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:02.445384 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-40bb78-29889960-98vl4" event={"ID":"a24ad108-152e-4690-82ae-71e98f901c61","Type":"ContainerDied","Data":"15263b3fdd4c8e0307a7b5b141ac324df2ce1c80cdca89ee758e4060d2cfc5fd"} Oct 30 22:00:02.445602 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:02.445587 4560 scope.go:122] "RemoveContainer" containerID="15263b3fdd4c8e0307a7b5b141ac324df2ce1c80cdca89ee758e4060d2cfc5fd" Oct 30 22:00:02.447019 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:02.447000 4560 generic.go:356] "Generic (PLEG): container finished" podID="818309d7-73a4-404f-9942-5bcbb2a4fd14" containerID="4d6bc6cba27f0c80e7d47f0c1cb7fb220755559ef9bd09b43c3740037ca939ed" exitCode=60 Oct 30 22:00:02.447115 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:02.447074 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889960-tfsx2" event={"ID":"818309d7-73a4-404f-9942-5bcbb2a4fd14","Type":"ContainerDied","Data":"4d6bc6cba27f0c80e7d47f0c1cb7fb220755559ef9bd09b43c3740037ca939ed"} Oct 30 22:00:02.447223 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:02.447210 4560 scope.go:122] "RemoveContainer" containerID="4d6bc6cba27f0c80e7d47f0c1cb7fb220755559ef9bd09b43c3740037ca939ed" Oct 30 22:00:02.448656 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:02.448627 4560 generic.go:356] "Generic (PLEG): container finished" podID="aa5fbac8-0bd8-4793-bbaa-c8dae197566d" containerID="87d7d3ae184aa97f41583624a8ec2801f637b3ca34c5e552b0b7b7b550eebd57" exitCode=60 Oct 30 22:00:02.448743 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:02.448704 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19dd4f-29889960-hcm42" event={"ID":"aa5fbac8-0bd8-4793-bbaa-c8dae197566d","Type":"ContainerDied","Data":"87d7d3ae184aa97f41583624a8ec2801f637b3ca34c5e552b0b7b7b550eebd57"} Oct 30 22:00:02.448863 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:02.448847 4560 scope.go:122] "RemoveContainer" containerID="87d7d3ae184aa97f41583624a8ec2801f637b3ca34c5e552b0b7b7b550eebd57" Oct 30 22:00:02.450860 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:02.450755 4560 generic.go:356] "Generic (PLEG): container finished" podID="ae04a1f9-8997-42e2-b7e9-94e84fd97d71" containerID="59f0cc17d8115ead00386ce4e0e717ff6f47a834e7efcea2b28f2765c6bee375" exitCode=60 Oct 30 22:00:02.450860 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:02.450839 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-bzl5x" event={"ID":"ae04a1f9-8997-42e2-b7e9-94e84fd97d71","Type":"ContainerDied","Data":"59f0cc17d8115ead00386ce4e0e717ff6f47a834e7efcea2b28f2765c6bee375"} Oct 30 22:00:02.451086 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:02.451063 4560 scope.go:122] "RemoveContainer" containerID="59f0cc17d8115ead00386ce4e0e717ff6f47a834e7efcea2b28f2765c6bee375" Oct 30 22:00:02.452969 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:02.452948 4560 generic.go:356] "Generic (PLEG): container finished" podID="d8a90e7a-b379-4742-a167-01aabd410bf9" containerID="8bb1a9ba9ccf6440e83e126274347df91ff3f912123229eed25f7cc691f8b89d" exitCode=60 Oct 30 22:00:02.453068 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:02.453010 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ca0a55-29889960-8wsx6" event={"ID":"d8a90e7a-b379-4742-a167-01aabd410bf9","Type":"ContainerDied","Data":"8bb1a9ba9ccf6440e83e126274347df91ff3f912123229eed25f7cc691f8b89d"} Oct 30 22:00:02.453210 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:02.453195 4560 scope.go:122] "RemoveContainer" containerID="8bb1a9ba9ccf6440e83e126274347df91ff3f912123229eed25f7cc691f8b89d" Oct 30 22:00:02.454788 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:02.454702 4560 generic.go:356] "Generic (PLEG): container finished" podID="22606928-4bc3-49e5-91e9-00497b022b10" containerID="9ff067a19a3e987be555c7047c75682154f8592b70ec1b8d28fc86f60dbb01a8" exitCode=60 Oct 30 22:00:02.454788 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:02.454749 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bfdf30b9-29889960-4c9wb" event={"ID":"22606928-4bc3-49e5-91e9-00497b022b10","Type":"ContainerDied","Data":"9ff067a19a3e987be555c7047c75682154f8592b70ec1b8d28fc86f60dbb01a8"} Oct 30 22:00:02.455009 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:02.454992 4560 scope.go:122] "RemoveContainer" containerID="9ff067a19a3e987be555c7047c75682154f8592b70ec1b8d28fc86f60dbb01a8" Oct 30 22:00:03.464021 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:03.463985 4560 generic.go:356] "Generic (PLEG): container finished" podID="08796630-f181-4b69-bd3c-32f627dff3ac" containerID="30152c6086bd8117f44d7b8f585d4d808f78da845273f76bc24213b6ea936c32" exitCode=60 Oct 30 22:00:03.464543 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:03.464068 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2df7b6-29889960-96qhk" event={"ID":"08796630-f181-4b69-bd3c-32f627dff3ac","Type":"ContainerDied","Data":"30152c6086bd8117f44d7b8f585d4d808f78da845273f76bc24213b6ea936c32"} Oct 30 22:00:03.464543 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:03.464188 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2df7b6-29889960-96qhk" event={"ID":"08796630-f181-4b69-bd3c-32f627dff3ac","Type":"ContainerStarted","Data":"75bc772204d89e6085ef8af9a5a5936ea3951f0f2623987ef15fb4ca27850daf"} Oct 30 22:00:03.466164 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:03.466135 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ed6135-29889960-bfnmd" event={"ID":"d29a3644-c4e8-40eb-a34b-f531c2a1565b","Type":"ContainerStarted","Data":"00f4fb6ff34433cfd34875df31fdca4f236c405ab95752939c3555ba243010b6"} Oct 30 22:00:03.467976 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:03.467951 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889960-vqpcn" event={"ID":"e174ecd3-9e50-458b-85aa-b130ceba6d70","Type":"ContainerStarted","Data":"638b6de6d5467077ed58a6a006c3f5ac2225c5e8f045711425d695058ae56210"} Oct 30 22:00:03.469810 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:03.469787 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-40bb78-29889960-98vl4" event={"ID":"a24ad108-152e-4690-82ae-71e98f901c61","Type":"ContainerStarted","Data":"c24a3ceba42c926ab2a75bc1a1fb8d1d5c740c87d5b71335f0bd11b2a4f75ec3"} Oct 30 22:00:03.471657 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:03.471634 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889960-tfsx2" event={"ID":"818309d7-73a4-404f-9942-5bcbb2a4fd14","Type":"ContainerStarted","Data":"f77a5a08563c00cb4cb2825d1590f04fafa9b23c5b69492e0ddf8f2f916f22a1"} Oct 30 22:00:03.473519 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:03.473495 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19dd4f-29889960-hcm42" event={"ID":"aa5fbac8-0bd8-4793-bbaa-c8dae197566d","Type":"ContainerStarted","Data":"517cf60d1f265380c91e0f76fe2e077f2346d19ef39b910e52ccf1c356472882"} Oct 30 22:00:03.475248 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:03.475227 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-bzl5x" event={"ID":"ae04a1f9-8997-42e2-b7e9-94e84fd97d71","Type":"ContainerStarted","Data":"e87c29eacbd59127c77ea19b5e354c90b868c2d59b49741f1b36a19dc991bc87"} Oct 30 22:00:03.477204 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:03.477184 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ca0a55-29889960-8wsx6" event={"ID":"d8a90e7a-b379-4742-a167-01aabd410bf9","Type":"ContainerStarted","Data":"530c1b6ba227741dcd0858722dbd45d78df6331d8e2d85276ace8307da4653e2"} Oct 30 22:00:03.479053 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:03.479036 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bfdf30b9-29889960-4c9wb" event={"ID":"22606928-4bc3-49e5-91e9-00497b022b10","Type":"ContainerStarted","Data":"7e35cba6c2c2eefb871732dc86019ac43af6a8dff90923cd4959be7bbee1ed2e"} Oct 30 22:00:03.480891 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:03.480861 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889960-g9zpz" event={"ID":"45e6d264-3503-493e-95f2-6308e04e93ab","Type":"ContainerStarted","Data":"21e5282fcca578aa6c4ac1207c6d6ee96e54dfb9405886ce607d46c70169119a"} Oct 30 22:00:03.837219 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:03.837171 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889960-vqpcn" podStartSLOduration=3.837155123 podStartE2EDuration="3.837155123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:03.835088057 +0000 UTC m=+2779.389787836" watchObservedRunningTime="2026-10-30 22:00:03.837155123 +0000 UTC m=+2779.391854900" Oct 30 22:00:03.956399 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:03.956330 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ed6135-29889960-bfnmd" podStartSLOduration=3.956314656 podStartE2EDuration="3.956314656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:03.956166581 +0000 UTC m=+2779.510866358" watchObservedRunningTime="2026-10-30 22:00:03.956314656 +0000 UTC m=+2779.511014434" Oct 30 22:00:04.196866 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.196765 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889960-g9zpz" podStartSLOduration=4.196751412 podStartE2EDuration="4.196751412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:04.194632652 +0000 UTC m=+2779.749332430" watchObservedRunningTime="2026-10-30 22:00:04.196751412 +0000 UTC m=+2779.751451189" Oct 30 22:00:04.309900 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.309844 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ca0a55-29889960-8wsx6" podStartSLOduration=4.309828517 podStartE2EDuration="4.309828517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:04.306074242 +0000 UTC m=+2779.860774019" watchObservedRunningTime="2026-10-30 22:00:04.309828517 +0000 UTC m=+2779.864528294" Oct 30 22:00:04.426549 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.426494 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19dd4f-29889960-hcm42" podStartSLOduration=4.426481092 podStartE2EDuration="4.426481092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:04.425733315 +0000 UTC m=+2779.980433095" watchObservedRunningTime="2026-10-30 22:00:04.426481092 +0000 UTC m=+2779.981180869" Oct 30 22:00:04.489666 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.489636 4560 generic.go:356] "Generic (PLEG): container finished" podID="45e6d264-3503-493e-95f2-6308e04e93ab" containerID="21e5282fcca578aa6c4ac1207c6d6ee96e54dfb9405886ce607d46c70169119a" exitCode=60 Oct 30 22:00:04.490120 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.489724 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889960-g9zpz" event={"ID":"45e6d264-3503-493e-95f2-6308e04e93ab","Type":"ContainerDied","Data":"21e5282fcca578aa6c4ac1207c6d6ee96e54dfb9405886ce607d46c70169119a"} Oct 30 22:00:04.490120 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.489796 4560 scope.go:122] "RemoveContainer" containerID="15ffa9ae5d631bbc0cbb9b7225ce4a22dc0e0d22691ee1046d53ee0d38731298" Oct 30 22:00:04.490120 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.489972 4560 scope.go:122] "RemoveContainer" containerID="21e5282fcca578aa6c4ac1207c6d6ee96e54dfb9405886ce607d46c70169119a" Oct 30 22:00:04.490303 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:00:04.490279 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-71aab5-29889960-g9zpz_odh-ai-gateway-infra(45e6d264-3503-493e-95f2-6308e04e93ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889960-g9zpz" podUID="45e6d264-3503-493e-95f2-6308e04e93ab" Oct 30 22:00:04.493085 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.493061 4560 generic.go:356] "Generic (PLEG): container finished" podID="08796630-f181-4b69-bd3c-32f627dff3ac" containerID="75bc772204d89e6085ef8af9a5a5936ea3951f0f2623987ef15fb4ca27850daf" exitCode=60 Oct 30 22:00:04.493193 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.493140 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2df7b6-29889960-96qhk" event={"ID":"08796630-f181-4b69-bd3c-32f627dff3ac","Type":"ContainerDied","Data":"75bc772204d89e6085ef8af9a5a5936ea3951f0f2623987ef15fb4ca27850daf"} Oct 30 22:00:04.493551 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.493530 4560 scope.go:122] "RemoveContainer" containerID="75bc772204d89e6085ef8af9a5a5936ea3951f0f2623987ef15fb4ca27850daf" Oct 30 22:00:04.493850 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:00:04.493830 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-2df7b6-29889960-96qhk_odh-ai-gateway-infra(08796630-f181-4b69-bd3c-32f627dff3ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2df7b6-29889960-96qhk" podUID="08796630-f181-4b69-bd3c-32f627dff3ac" Oct 30 22:00:04.495819 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.495798 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-40bb78-6d7d4d7644-jwwbv"] Oct 30 22:00:04.495945 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.495849 4560 generic.go:356] "Generic (PLEG): container finished" podID="d29a3644-c4e8-40eb-a34b-f531c2a1565b" containerID="00f4fb6ff34433cfd34875df31fdca4f236c405ab95752939c3555ba243010b6" exitCode=60 Oct 30 22:00:04.495945 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.495854 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ed6135-29889960-bfnmd" event={"ID":"d29a3644-c4e8-40eb-a34b-f531c2a1565b","Type":"ContainerDied","Data":"00f4fb6ff34433cfd34875df31fdca4f236c405ab95752939c3555ba243010b6"} Oct 30 22:00:04.496235 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.496220 4560 scope.go:122] "RemoveContainer" containerID="00f4fb6ff34433cfd34875df31fdca4f236c405ab95752939c3555ba243010b6" Oct 30 22:00:04.496525 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:00:04.496497 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-ed6135-29889960-bfnmd_odh-ai-gateway-infra(d29a3644-c4e8-40eb-a34b-f531c2a1565b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ed6135-29889960-bfnmd" podUID="d29a3644-c4e8-40eb-a34b-f531c2a1565b" Oct 30 22:00:04.497780 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.497761 4560 generic.go:356] "Generic (PLEG): container finished" podID="e174ecd3-9e50-458b-85aa-b130ceba6d70" containerID="638b6de6d5467077ed58a6a006c3f5ac2225c5e8f045711425d695058ae56210" exitCode=60 Oct 30 22:00:04.497863 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.497839 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889960-vqpcn" event={"ID":"e174ecd3-9e50-458b-85aa-b130ceba6d70","Type":"ContainerDied","Data":"638b6de6d5467077ed58a6a006c3f5ac2225c5e8f045711425d695058ae56210"} Oct 30 22:00:04.498014 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.498001 4560 scope.go:122] "RemoveContainer" containerID="638b6de6d5467077ed58a6a006c3f5ac2225c5e8f045711425d695058ae56210" Oct 30 22:00:04.498219 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:00:04.498195 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889960-vqpcn_odh-ai-gateway-infra(e174ecd3-9e50-458b-85aa-b130ceba6d70)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889960-vqpcn" podUID="e174ecd3-9e50-458b-85aa-b130ceba6d70" Oct 30 22:00:04.499643 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.499625 4560 generic.go:356] "Generic (PLEG): container finished" podID="a24ad108-152e-4690-82ae-71e98f901c61" containerID="c24a3ceba42c926ab2a75bc1a1fb8d1d5c740c87d5b71335f0bd11b2a4f75ec3" exitCode=60 Oct 30 22:00:04.499730 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.499707 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-40bb78-29889960-98vl4" event={"ID":"a24ad108-152e-4690-82ae-71e98f901c61","Type":"ContainerDied","Data":"c24a3ceba42c926ab2a75bc1a1fb8d1d5c740c87d5b71335f0bd11b2a4f75ec3"} Oct 30 22:00:04.499868 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.499845 4560 scope.go:122] "RemoveContainer" containerID="c24a3ceba42c926ab2a75bc1a1fb8d1d5c740c87d5b71335f0bd11b2a4f75ec3" Oct 30 22:00:04.500062 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:00:04.500041 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-40bb78-29889960-98vl4_odh-ai-gateway-infra(a24ad108-152e-4690-82ae-71e98f901c61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-40bb78-29889960-98vl4" podUID="a24ad108-152e-4690-82ae-71e98f901c61" Oct 30 22:00:04.501520 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.501498 4560 generic.go:356] "Generic (PLEG): container finished" podID="818309d7-73a4-404f-9942-5bcbb2a4fd14" containerID="f77a5a08563c00cb4cb2825d1590f04fafa9b23c5b69492e0ddf8f2f916f22a1" exitCode=60 Oct 30 22:00:04.501604 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.501561 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889960-tfsx2" event={"ID":"818309d7-73a4-404f-9942-5bcbb2a4fd14","Type":"ContainerDied","Data":"f77a5a08563c00cb4cb2825d1590f04fafa9b23c5b69492e0ddf8f2f916f22a1"} Oct 30 22:00:04.501796 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.501770 4560 scope.go:122] "RemoveContainer" containerID="f77a5a08563c00cb4cb2825d1590f04fafa9b23c5b69492e0ddf8f2f916f22a1" Oct 30 22:00:04.502216 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:00:04.502180 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-554f46-29889960-tfsx2_odh-ai-gateway-infra(818309d7-73a4-404f-9942-5bcbb2a4fd14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889960-tfsx2" podUID="818309d7-73a4-404f-9942-5bcbb2a4fd14" Oct 30 22:00:04.503615 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.503595 4560 generic.go:356] "Generic (PLEG): container finished" podID="aa5fbac8-0bd8-4793-bbaa-c8dae197566d" containerID="517cf60d1f265380c91e0f76fe2e077f2346d19ef39b910e52ccf1c356472882" exitCode=60 Oct 30 22:00:04.503700 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.503663 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19dd4f-29889960-hcm42" event={"ID":"aa5fbac8-0bd8-4793-bbaa-c8dae197566d","Type":"ContainerDied","Data":"517cf60d1f265380c91e0f76fe2e077f2346d19ef39b910e52ccf1c356472882"} Oct 30 22:00:04.503868 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.503855 4560 scope.go:122] "RemoveContainer" containerID="517cf60d1f265380c91e0f76fe2e077f2346d19ef39b910e52ccf1c356472882" Oct 30 22:00:04.504053 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:00:04.504028 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-19dd4f-29889960-hcm42_odh-ai-gateway-infra(aa5fbac8-0bd8-4793-bbaa-c8dae197566d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19dd4f-29889960-hcm42" podUID="aa5fbac8-0bd8-4793-bbaa-c8dae197566d" Oct 30 22:00:04.504638 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.504624 4560 scope.go:122] "RemoveContainer" containerID="30152c6086bd8117f44d7b8f585d4d808f78da845273f76bc24213b6ea936c32" Oct 30 22:00:04.505573 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.505557 4560 generic.go:356] "Generic (PLEG): container finished" podID="ae04a1f9-8997-42e2-b7e9-94e84fd97d71" containerID="e87c29eacbd59127c77ea19b5e354c90b868c2d59b49741f1b36a19dc991bc87" exitCode=60 Oct 30 22:00:04.505631 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.505618 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-bzl5x" event={"ID":"ae04a1f9-8997-42e2-b7e9-94e84fd97d71","Type":"ContainerDied","Data":"e87c29eacbd59127c77ea19b5e354c90b868c2d59b49741f1b36a19dc991bc87"} Oct 30 22:00:04.505820 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.505805 4560 scope.go:122] "RemoveContainer" containerID="e87c29eacbd59127c77ea19b5e354c90b868c2d59b49741f1b36a19dc991bc87" Oct 30 22:00:04.506025 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:00:04.505998 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889960-bzl5x_odh-ai-gateway-infra(ae04a1f9-8997-42e2-b7e9-94e84fd97d71)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-bzl5x" podUID="ae04a1f9-8997-42e2-b7e9-94e84fd97d71" Oct 30 22:00:04.507273 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.507253 4560 generic.go:356] "Generic (PLEG): container finished" podID="d8a90e7a-b379-4742-a167-01aabd410bf9" containerID="530c1b6ba227741dcd0858722dbd45d78df6331d8e2d85276ace8307da4653e2" exitCode=60 Oct 30 22:00:04.507369 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.507352 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ca0a55-29889960-8wsx6" event={"ID":"d8a90e7a-b379-4742-a167-01aabd410bf9","Type":"ContainerDied","Data":"530c1b6ba227741dcd0858722dbd45d78df6331d8e2d85276ace8307da4653e2"} Oct 30 22:00:04.507499 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.507485 4560 scope.go:122] "RemoveContainer" containerID="530c1b6ba227741dcd0858722dbd45d78df6331d8e2d85276ace8307da4653e2" Oct 30 22:00:04.507724 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:00:04.507706 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-ca0a55-29889960-8wsx6_odh-ai-gateway-infra(d8a90e7a-b379-4742-a167-01aabd410bf9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ca0a55-29889960-8wsx6" podUID="d8a90e7a-b379-4742-a167-01aabd410bf9" Oct 30 22:00:04.508978 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.508958 4560 generic.go:356] "Generic (PLEG): container finished" podID="22606928-4bc3-49e5-91e9-00497b022b10" containerID="7e35cba6c2c2eefb871732dc86019ac43af6a8dff90923cd4959be7bbee1ed2e" exitCode=60 Oct 30 22:00:04.509070 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.508998 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bfdf30b9-29889960-4c9wb" event={"ID":"22606928-4bc3-49e5-91e9-00497b022b10","Type":"ContainerDied","Data":"7e35cba6c2c2eefb871732dc86019ac43af6a8dff90923cd4959be7bbee1ed2e"} Oct 30 22:00:04.509215 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.509199 4560 scope.go:122] "RemoveContainer" containerID="7e35cba6c2c2eefb871732dc86019ac43af6a8dff90923cd4959be7bbee1ed2e" Oct 30 22:00:04.509410 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:00:04.509392 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-bfdf30b9-29889960-4c9wb_odh-ai-gateway-infra(22606928-4bc3-49e5-91e9-00497b022b10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bfdf30b9-29889960-4c9wb" podUID="22606928-4bc3-49e5-91e9-00497b022b10" Oct 30 22:00:04.517211 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.517198 4560 scope.go:122] "RemoveContainer" containerID="c111b5384126890cef4f2c52680980aebc3cc4548777d0a2cc71c5b5e03f0e9a" Oct 30 22:00:04.528966 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.528946 4560 scope.go:122] "RemoveContainer" containerID="edb3d28e5734985e8d7b1dfa0ac4090cd33d2bcb8ff90d28568c7135a929339f" Oct 30 22:00:04.540757 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.540716 4560 scope.go:122] "RemoveContainer" containerID="15263b3fdd4c8e0307a7b5b141ac324df2ce1c80cdca89ee758e4060d2cfc5fd" Oct 30 22:00:04.552849 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.552833 4560 scope.go:122] "RemoveContainer" containerID="4d6bc6cba27f0c80e7d47f0c1cb7fb220755559ef9bd09b43c3740037ca939ed" Oct 30 22:00:04.557251 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.557213 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bfdf30b9-29889960-4c9wb" podStartSLOduration=4.557202282 podStartE2EDuration="4.557202282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:04.554872092 +0000 UTC m=+2780.109571869" watchObservedRunningTime="2026-10-30 22:00:04.557202282 +0000 UTC m=+2780.111902058" Oct 30 22:00:04.565032 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.565017 4560 scope.go:122] "RemoveContainer" containerID="87d7d3ae184aa97f41583624a8ec2801f637b3ca34c5e552b0b7b7b550eebd57" Oct 30 22:00:04.577828 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.577808 4560 scope.go:122] "RemoveContainer" containerID="59f0cc17d8115ead00386ce4e0e717ff6f47a834e7efcea2b28f2765c6bee375" Oct 30 22:00:04.589747 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.589725 4560 scope.go:122] "RemoveContainer" containerID="8bb1a9ba9ccf6440e83e126274347df91ff3f912123229eed25f7cc691f8b89d" Oct 30 22:00:04.601472 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.601451 4560 scope.go:122] "RemoveContainer" containerID="9ff067a19a3e987be555c7047c75682154f8592b70ec1b8d28fc86f60dbb01a8" Oct 30 22:00:04.686448 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.686396 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889960-tfsx2" podStartSLOduration=4.686382159 podStartE2EDuration="4.686382159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:04.685012431 +0000 UTC m=+2780.239712210" watchObservedRunningTime="2026-10-30 22:00:04.686382159 +0000 UTC m=+2780.241081935" Oct 30 22:00:04.805075 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:04.804959 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2df7b6-29889960-96qhk" podStartSLOduration=4.804945185 podStartE2EDuration="4.804945185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:04.80116628 +0000 UTC m=+2780.355866071" watchObservedRunningTime="2026-10-30 22:00:04.804945185 +0000 UTC m=+2780.359644962" Oct 30 22:00:05.140575 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.140455 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-bzl5x" podStartSLOduration=5.140438068 podStartE2EDuration="5.140438068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:04.972516358 +0000 UTC m=+2780.527216135" watchObservedRunningTime="2026-10-30 22:00:05.140438068 +0000 UTC m=+2780.695137846" Oct 30 22:00:05.521034 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.521000 4560 scope.go:122] "RemoveContainer" containerID="7e35cba6c2c2eefb871732dc86019ac43af6a8dff90923cd4959be7bbee1ed2e" Oct 30 22:00:05.521511 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:00:05.521321 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-bfdf30b9-29889960-4c9wb_odh-ai-gateway-infra(22606928-4bc3-49e5-91e9-00497b022b10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bfdf30b9-29889960-4c9wb" podUID="22606928-4bc3-49e5-91e9-00497b022b10" Oct 30 22:00:05.522731 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.522710 4560 scope.go:122] "RemoveContainer" containerID="21e5282fcca578aa6c4ac1207c6d6ee96e54dfb9405886ce607d46c70169119a" Oct 30 22:00:05.523007 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:00:05.522987 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-71aab5-29889960-g9zpz_odh-ai-gateway-infra(45e6d264-3503-493e-95f2-6308e04e93ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889960-g9zpz" podUID="45e6d264-3503-493e-95f2-6308e04e93ab" Oct 30 22:00:05.524256 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.524234 4560 scope.go:122] "RemoveContainer" containerID="75bc772204d89e6085ef8af9a5a5936ea3951f0f2623987ef15fb4ca27850daf" Oct 30 22:00:05.524532 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:00:05.524507 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-2df7b6-29889960-96qhk_odh-ai-gateway-infra(08796630-f181-4b69-bd3c-32f627dff3ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2df7b6-29889960-96qhk" podUID="08796630-f181-4b69-bd3c-32f627dff3ac" Oct 30 22:00:05.525789 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.525771 4560 scope.go:122] "RemoveContainer" containerID="00f4fb6ff34433cfd34875df31fdca4f236c405ab95752939c3555ba243010b6" Oct 30 22:00:05.526029 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:00:05.526002 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-ed6135-29889960-bfnmd_odh-ai-gateway-infra(d29a3644-c4e8-40eb-a34b-f531c2a1565b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ed6135-29889960-bfnmd" podUID="d29a3644-c4e8-40eb-a34b-f531c2a1565b" Oct 30 22:00:05.527423 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.527405 4560 scope.go:122] "RemoveContainer" containerID="638b6de6d5467077ed58a6a006c3f5ac2225c5e8f045711425d695058ae56210" Oct 30 22:00:05.527627 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:00:05.527608 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889960-vqpcn_odh-ai-gateway-infra(e174ecd3-9e50-458b-85aa-b130ceba6d70)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889960-vqpcn" podUID="e174ecd3-9e50-458b-85aa-b130ceba6d70" Oct 30 22:00:05.528763 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.528745 4560 scope.go:122] "RemoveContainer" containerID="c24a3ceba42c926ab2a75bc1a1fb8d1d5c740c87d5b71335f0bd11b2a4f75ec3" Oct 30 22:00:05.528932 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:00:05.528911 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-40bb78-29889960-98vl4_odh-ai-gateway-infra(a24ad108-152e-4690-82ae-71e98f901c61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-40bb78-29889960-98vl4" podUID="a24ad108-152e-4690-82ae-71e98f901c61" Oct 30 22:00:05.530208 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.530193 4560 scope.go:122] "RemoveContainer" containerID="f77a5a08563c00cb4cb2825d1590f04fafa9b23c5b69492e0ddf8f2f916f22a1" Oct 30 22:00:05.530400 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:00:05.530383 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-554f46-29889960-tfsx2_odh-ai-gateway-infra(818309d7-73a4-404f-9942-5bcbb2a4fd14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889960-tfsx2" podUID="818309d7-73a4-404f-9942-5bcbb2a4fd14" Oct 30 22:00:05.531845 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.531816 4560 scope.go:122] "RemoveContainer" containerID="517cf60d1f265380c91e0f76fe2e077f2346d19ef39b910e52ccf1c356472882" Oct 30 22:00:05.532279 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:00:05.532208 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-19dd4f-29889960-hcm42_odh-ai-gateway-infra(aa5fbac8-0bd8-4793-bbaa-c8dae197566d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19dd4f-29889960-hcm42" podUID="aa5fbac8-0bd8-4793-bbaa-c8dae197566d" Oct 30 22:00:05.534980 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.534959 4560 scope.go:122] "RemoveContainer" containerID="e87c29eacbd59127c77ea19b5e354c90b868c2d59b49741f1b36a19dc991bc87" Oct 30 22:00:05.535183 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:00:05.535162 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889960-bzl5x_odh-ai-gateway-infra(ae04a1f9-8997-42e2-b7e9-94e84fd97d71)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-bzl5x" podUID="ae04a1f9-8997-42e2-b7e9-94e84fd97d71" Oct 30 22:00:05.535906 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.535886 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf"] Oct 30 22:00:05.536457 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.536440 4560 scope.go:122] "RemoveContainer" containerID="530c1b6ba227741dcd0858722dbd45d78df6331d8e2d85276ace8307da4653e2" Oct 30 22:00:05.536613 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:00:05.536597 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-ca0a55-29889960-8wsx6_odh-ai-gateway-infra(d8a90e7a-b379-4742-a167-01aabd410bf9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ca0a55-29889960-8wsx6" podUID="d8a90e7a-b379-4742-a167-01aabd410bf9" Oct 30 22:00:05.541075 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.541060 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" Oct 30 22:00:05.593775 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.593748 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-bfdf30b9-openshift-default-dockercfg-pzmxv\"" Oct 30 22:00:05.601661 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.601634 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf"] Oct 30 22:00:05.604108 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.604076 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/156f466e-16e0-441c-9ce5-796acd8551b2-istio-podinfo\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf\" (UID: \"156f466e-16e0-441c-9ce5-796acd8551b2\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" Oct 30 22:00:05.604272 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.604238 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/156f466e-16e0-441c-9ce5-796acd8551b2-workload-certs\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf\" (UID: \"156f466e-16e0-441c-9ce5-796acd8551b2\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" Oct 30 22:00:05.604376 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.604303 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/156f466e-16e0-441c-9ce5-796acd8551b2-istio-token\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf\" (UID: \"156f466e-16e0-441c-9ce5-796acd8551b2\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" Oct 30 22:00:05.604606 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.604582 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/156f466e-16e0-441c-9ce5-796acd8551b2-istio-data\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf\" (UID: \"156f466e-16e0-441c-9ce5-796acd8551b2\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" Oct 30 22:00:05.604705 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.604659 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/156f466e-16e0-441c-9ce5-796acd8551b2-istio-envoy\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf\" (UID: \"156f466e-16e0-441c-9ce5-796acd8551b2\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" Oct 30 22:00:05.604810 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.604793 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/156f466e-16e0-441c-9ce5-796acd8551b2-istiod-ca-cert\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf\" (UID: \"156f466e-16e0-441c-9ce5-796acd8551b2\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" Oct 30 22:00:05.604878 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.604827 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf4mg\" (UniqueName: \"kubernetes.io/projected/156f466e-16e0-441c-9ce5-796acd8551b2-kube-api-access-sf4mg\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf\" (UID: \"156f466e-16e0-441c-9ce5-796acd8551b2\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" Oct 30 22:00:05.604935 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.604921 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/156f466e-16e0-441c-9ce5-796acd8551b2-workload-socket\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf\" (UID: \"156f466e-16e0-441c-9ce5-796acd8551b2\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" Oct 30 22:00:05.604989 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.604951 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/156f466e-16e0-441c-9ce5-796acd8551b2-credential-socket\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf\" (UID: \"156f466e-16e0-441c-9ce5-796acd8551b2\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" Oct 30 22:00:05.706265 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.706223 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/156f466e-16e0-441c-9ce5-796acd8551b2-istio-podinfo\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf\" (UID: \"156f466e-16e0-441c-9ce5-796acd8551b2\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" Oct 30 22:00:05.706498 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.706285 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/156f466e-16e0-441c-9ce5-796acd8551b2-workload-certs\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf\" (UID: \"156f466e-16e0-441c-9ce5-796acd8551b2\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" Oct 30 22:00:05.706498 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.706317 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/156f466e-16e0-441c-9ce5-796acd8551b2-istio-token\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf\" (UID: \"156f466e-16e0-441c-9ce5-796acd8551b2\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" Oct 30 22:00:05.706498 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.706353 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/156f466e-16e0-441c-9ce5-796acd8551b2-istio-data\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf\" (UID: \"156f466e-16e0-441c-9ce5-796acd8551b2\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" Oct 30 22:00:05.706498 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.706392 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/156f466e-16e0-441c-9ce5-796acd8551b2-istio-envoy\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf\" (UID: \"156f466e-16e0-441c-9ce5-796acd8551b2\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" Oct 30 22:00:05.706498 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.706448 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/156f466e-16e0-441c-9ce5-796acd8551b2-istiod-ca-cert\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf\" (UID: \"156f466e-16e0-441c-9ce5-796acd8551b2\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" Oct 30 22:00:05.706498 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.706475 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf4mg\" (UniqueName: \"kubernetes.io/projected/156f466e-16e0-441c-9ce5-796acd8551b2-kube-api-access-sf4mg\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf\" (UID: \"156f466e-16e0-441c-9ce5-796acd8551b2\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" Oct 30 22:00:05.706842 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.706535 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/156f466e-16e0-441c-9ce5-796acd8551b2-workload-socket\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf\" (UID: \"156f466e-16e0-441c-9ce5-796acd8551b2\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" Oct 30 22:00:05.706842 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.706563 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/156f466e-16e0-441c-9ce5-796acd8551b2-credential-socket\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf\" (UID: \"156f466e-16e0-441c-9ce5-796acd8551b2\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" Oct 30 22:00:05.707024 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.706990 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/156f466e-16e0-441c-9ce5-796acd8551b2-workload-certs\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf\" (UID: \"156f466e-16e0-441c-9ce5-796acd8551b2\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" Oct 30 22:00:05.707334 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.707305 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/156f466e-16e0-441c-9ce5-796acd8551b2-credential-socket\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf\" (UID: \"156f466e-16e0-441c-9ce5-796acd8551b2\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" Oct 30 22:00:05.707566 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.707537 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/156f466e-16e0-441c-9ce5-796acd8551b2-istio-data\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf\" (UID: \"156f466e-16e0-441c-9ce5-796acd8551b2\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" Oct 30 22:00:05.707665 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.707645 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/156f466e-16e0-441c-9ce5-796acd8551b2-istiod-ca-cert\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf\" (UID: \"156f466e-16e0-441c-9ce5-796acd8551b2\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" Oct 30 22:00:05.707742 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.707682 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/156f466e-16e0-441c-9ce5-796acd8551b2-workload-socket\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf\" (UID: \"156f466e-16e0-441c-9ce5-796acd8551b2\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" Oct 30 22:00:05.709015 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.708994 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/156f466e-16e0-441c-9ce5-796acd8551b2-istio-envoy\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf\" (UID: \"156f466e-16e0-441c-9ce5-796acd8551b2\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" Oct 30 22:00:05.709122 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.709041 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/156f466e-16e0-441c-9ce5-796acd8551b2-istio-podinfo\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf\" (UID: \"156f466e-16e0-441c-9ce5-796acd8551b2\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" Oct 30 22:00:05.783132 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.783059 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf4mg\" (UniqueName: \"kubernetes.io/projected/156f466e-16e0-441c-9ce5-796acd8551b2-kube-api-access-sf4mg\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf\" (UID: \"156f466e-16e0-441c-9ce5-796acd8551b2\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" Oct 30 22:00:05.784562 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.784542 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/156f466e-16e0-441c-9ce5-796acd8551b2-istio-token\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf\" (UID: \"156f466e-16e0-441c-9ce5-796acd8551b2\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" Oct 30 22:00:05.854953 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:05.854922 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" Oct 30 22:00:06.338623 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:06.337733 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf"] Oct 30 22:00:06.356597 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:06.351460 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:00:06.356597 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:06.351550 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:00:06.356597 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:06.351598 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:00:06.559969 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:06.559931 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" event={"ID":"156f466e-16e0-441c-9ce5-796acd8551b2","Type":"ContainerStarted","Data":"1ba3e97851e342b02fe94b46a8d3de3994e2840360a6cb9de76b226ec01f4a27"} Oct 30 22:00:06.560511 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:06.560011 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" event={"ID":"156f466e-16e0-441c-9ce5-796acd8551b2","Type":"ContainerStarted","Data":"b90ead70ad1c49325a286dcf50c0fcbbb31cd727417372ea820beed62032662c"} Oct 30 22:00:06.855061 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:06.855027 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" Oct 30 22:00:07.779404 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:07.779317 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" podStartSLOduration=2.779301253 podStartE2EDuration="2.779301253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:07.777459465 +0000 UTC m=+2783.332159244" watchObservedRunningTime="2026-10-30 22:00:07.779301253 +0000 UTC m=+2783.334001093" Oct 30 22:00:07.857635 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:07.857597 4560 patch_prober.go:29] interesting pod/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 22:00:07.857814 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:07.857656 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" podUID="156f466e-16e0-441c-9ce5-796acd8551b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 22:00:08.855635 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:08.855598 4560 patch_prober.go:29] interesting pod/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 22:00:08.856055 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:08.855652 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" podUID="156f466e-16e0-441c-9ce5-796acd8551b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 22:00:09.496948 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:09.496884 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-40bb78-6d7d4d7644-jwwbv" podUID="345a9ec6-062e-4402-83fc-15adac6800bd" containerName="payload-processing" containerID="cri-o://9e97a8c9bbf328276d7147f509d20b591fa5833094c6d49562c80de24c9ea5db" gracePeriod=25 Oct 30 22:00:09.615323 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:09.615178 4560 generic.go:356] "Generic (PLEG): container finished" podID="345a9ec6-062e-4402-83fc-15adac6800bd" containerID="9e97a8c9bbf328276d7147f509d20b591fa5833094c6d49562c80de24c9ea5db" exitCode=0 Oct 30 22:00:09.615323 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:09.615294 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-40bb78-6d7d4d7644-jwwbv" event={"ID":"345a9ec6-062e-4402-83fc-15adac6800bd","Type":"ContainerDied","Data":"9e97a8c9bbf328276d7147f509d20b591fa5833094c6d49562c80de24c9ea5db"} Oct 30 22:00:09.870307 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:09.864625 4560 patch_prober.go:29] interesting pod/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 22:00:09.870307 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:09.864688 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" podUID="156f466e-16e0-441c-9ce5-796acd8551b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 22:00:09.870307 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:09.866762 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-40bb78-6d7d4d7644-jwwbv" Oct 30 22:00:09.924751 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:09.924580 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/345a9ec6-062e-4402-83fc-15adac6800bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/345a9ec6-062e-4402-83fc-15adac6800bd-plugins-config-volume\") pod \"345a9ec6-062e-4402-83fc-15adac6800bd\" (UID: \"345a9ec6-062e-4402-83fc-15adac6800bd\") " Oct 30 22:00:09.924751 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:09.924682 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/345a9ec6-062e-4402-83fc-15adac6800bd-kube-api-access-p8bzp\" (UniqueName: \"kubernetes.io/projected/345a9ec6-062e-4402-83fc-15adac6800bd-kube-api-access-p8bzp\") pod \"345a9ec6-062e-4402-83fc-15adac6800bd\" (UID: \"345a9ec6-062e-4402-83fc-15adac6800bd\") " Oct 30 22:00:09.925058 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:09.925028 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/345a9ec6-062e-4402-83fc-15adac6800bd-plugins-config-volume" pod "345a9ec6-062e-4402-83fc-15adac6800bd" (UID: "345a9ec6-062e-4402-83fc-15adac6800bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:09.925241 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:09.925221 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/345a9ec6-062e-4402-83fc-15adac6800bd-plugins-config-volume\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:09.936555 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:09.936505 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/345a9ec6-062e-4402-83fc-15adac6800bd-kube-api-access-p8bzp" pod "345a9ec6-062e-4402-83fc-15adac6800bd" (UID: "345a9ec6-062e-4402-83fc-15adac6800bd"). InnerVolumeSpecName "kube-api-access-p8bzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:10.030258 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.030214 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8bzp\" (UniqueName: \"kubernetes.io/projected/345a9ec6-062e-4402-83fc-15adac6800bd-kube-api-access-p8bzp\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:10.496619 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.496588 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9"] Oct 30 22:00:10.497149 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.497128 4560 cpu_manager.go:414] "containerMap: removing container" podUID="345a9ec6-062e-4402-83fc-15adac6800bd" containerName="payload-processing" Oct 30 22:00:10.497149 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.497148 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="345a9ec6-062e-4402-83fc-15adac6800bd" containerName="payload-processing" Oct 30 22:00:10.497389 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.497288 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="345a9ec6-062e-4402-83fc-15adac6800bd" containerName="payload-processing" Oct 30 22:00:10.500786 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.500768 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" Oct 30 22:00:10.559370 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.555379 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-28c192-openshift-default-dockercfg-wbzb6\"" Oct 30 22:00:10.569857 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.566197 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f4405ac-93f4-4a1d-a335-5990238c4294-istio-data\") pod \"e2e-api-a-28c192-openshift-default-5586f598c-r8qq9\" (UID: \"7f4405ac-93f4-4a1d-a335-5990238c4294\") " pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" Oct 30 22:00:10.569857 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.566252 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f4405ac-93f4-4a1d-a335-5990238c4294-istio-podinfo\") pod \"e2e-api-a-28c192-openshift-default-5586f598c-r8qq9\" (UID: \"7f4405ac-93f4-4a1d-a335-5990238c4294\") " pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" Oct 30 22:00:10.569857 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.566293 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f4405ac-93f4-4a1d-a335-5990238c4294-istio-token\") pod \"e2e-api-a-28c192-openshift-default-5586f598c-r8qq9\" (UID: \"7f4405ac-93f4-4a1d-a335-5990238c4294\") " pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" Oct 30 22:00:10.569857 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.566365 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4405ac-93f4-4a1d-a335-5990238c4294-credential-socket\") pod \"e2e-api-a-28c192-openshift-default-5586f598c-r8qq9\" (UID: \"7f4405ac-93f4-4a1d-a335-5990238c4294\") " pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" Oct 30 22:00:10.569857 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.566438 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f4405ac-93f4-4a1d-a335-5990238c4294-workload-certs\") pod \"e2e-api-a-28c192-openshift-default-5586f598c-r8qq9\" (UID: \"7f4405ac-93f4-4a1d-a335-5990238c4294\") " pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" Oct 30 22:00:10.569857 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.566465 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4405ac-93f4-4a1d-a335-5990238c4294-workload-socket\") pod \"e2e-api-a-28c192-openshift-default-5586f598c-r8qq9\" (UID: \"7f4405ac-93f4-4a1d-a335-5990238c4294\") " pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" Oct 30 22:00:10.569857 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.566534 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jvln\" (UniqueName: \"kubernetes.io/projected/7f4405ac-93f4-4a1d-a335-5990238c4294-kube-api-access-7jvln\") pod \"e2e-api-a-28c192-openshift-default-5586f598c-r8qq9\" (UID: \"7f4405ac-93f4-4a1d-a335-5990238c4294\") " pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" Oct 30 22:00:10.569857 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.566572 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f4405ac-93f4-4a1d-a335-5990238c4294-istio-envoy\") pod \"e2e-api-a-28c192-openshift-default-5586f598c-r8qq9\" (UID: \"7f4405ac-93f4-4a1d-a335-5990238c4294\") " pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" Oct 30 22:00:10.569857 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.566620 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f4405ac-93f4-4a1d-a335-5990238c4294-istiod-ca-cert\") pod \"e2e-api-a-28c192-openshift-default-5586f598c-r8qq9\" (UID: \"7f4405ac-93f4-4a1d-a335-5990238c4294\") " pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" Oct 30 22:00:10.603975 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.603939 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9"] Oct 30 22:00:10.712362 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.711808 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f4405ac-93f4-4a1d-a335-5990238c4294-istiod-ca-cert\") pod \"e2e-api-a-28c192-openshift-default-5586f598c-r8qq9\" (UID: \"7f4405ac-93f4-4a1d-a335-5990238c4294\") " pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" Oct 30 22:00:10.712362 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.711949 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f4405ac-93f4-4a1d-a335-5990238c4294-istio-data\") pod \"e2e-api-a-28c192-openshift-default-5586f598c-r8qq9\" (UID: \"7f4405ac-93f4-4a1d-a335-5990238c4294\") " pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" Oct 30 22:00:10.712362 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.711980 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f4405ac-93f4-4a1d-a335-5990238c4294-istio-podinfo\") pod \"e2e-api-a-28c192-openshift-default-5586f598c-r8qq9\" (UID: \"7f4405ac-93f4-4a1d-a335-5990238c4294\") " pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" Oct 30 22:00:10.712362 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.712020 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f4405ac-93f4-4a1d-a335-5990238c4294-istio-token\") pod \"e2e-api-a-28c192-openshift-default-5586f598c-r8qq9\" (UID: \"7f4405ac-93f4-4a1d-a335-5990238c4294\") " pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" Oct 30 22:00:10.712362 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.712070 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4405ac-93f4-4a1d-a335-5990238c4294-credential-socket\") pod \"e2e-api-a-28c192-openshift-default-5586f598c-r8qq9\" (UID: \"7f4405ac-93f4-4a1d-a335-5990238c4294\") " pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" Oct 30 22:00:10.712362 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.712137 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f4405ac-93f4-4a1d-a335-5990238c4294-workload-certs\") pod \"e2e-api-a-28c192-openshift-default-5586f598c-r8qq9\" (UID: \"7f4405ac-93f4-4a1d-a335-5990238c4294\") " pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" Oct 30 22:00:10.712362 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.712165 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4405ac-93f4-4a1d-a335-5990238c4294-workload-socket\") pod \"e2e-api-a-28c192-openshift-default-5586f598c-r8qq9\" (UID: \"7f4405ac-93f4-4a1d-a335-5990238c4294\") " pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" Oct 30 22:00:10.712362 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.712233 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jvln\" (UniqueName: \"kubernetes.io/projected/7f4405ac-93f4-4a1d-a335-5990238c4294-kube-api-access-7jvln\") pod \"e2e-api-a-28c192-openshift-default-5586f598c-r8qq9\" (UID: \"7f4405ac-93f4-4a1d-a335-5990238c4294\") " pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" Oct 30 22:00:10.712362 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.712273 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f4405ac-93f4-4a1d-a335-5990238c4294-istio-envoy\") pod \"e2e-api-a-28c192-openshift-default-5586f598c-r8qq9\" (UID: \"7f4405ac-93f4-4a1d-a335-5990238c4294\") " pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" Oct 30 22:00:10.720366 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.715495 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f4405ac-93f4-4a1d-a335-5990238c4294-workload-certs\") pod \"e2e-api-a-28c192-openshift-default-5586f598c-r8qq9\" (UID: \"7f4405ac-93f4-4a1d-a335-5990238c4294\") " pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" Oct 30 22:00:10.720366 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.716236 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4405ac-93f4-4a1d-a335-5990238c4294-credential-socket\") pod \"e2e-api-a-28c192-openshift-default-5586f598c-r8qq9\" (UID: \"7f4405ac-93f4-4a1d-a335-5990238c4294\") " pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" Oct 30 22:00:10.720366 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.717217 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f4405ac-93f4-4a1d-a335-5990238c4294-istiod-ca-cert\") pod \"e2e-api-a-28c192-openshift-default-5586f598c-r8qq9\" (UID: \"7f4405ac-93f4-4a1d-a335-5990238c4294\") " pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" Oct 30 22:00:10.720366 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.717472 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f4405ac-93f4-4a1d-a335-5990238c4294-istio-data\") pod \"e2e-api-a-28c192-openshift-default-5586f598c-r8qq9\" (UID: \"7f4405ac-93f4-4a1d-a335-5990238c4294\") " pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" Oct 30 22:00:10.720366 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.718115 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4405ac-93f4-4a1d-a335-5990238c4294-workload-socket\") pod \"e2e-api-a-28c192-openshift-default-5586f598c-r8qq9\" (UID: \"7f4405ac-93f4-4a1d-a335-5990238c4294\") " pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" Oct 30 22:00:10.748426 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.748034 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-40bb78-6d7d4d7644-jwwbv" event={"ID":"345a9ec6-062e-4402-83fc-15adac6800bd","Type":"ContainerDied","Data":"50bab75277461bba0ebcf20d56fd438b8f32de22f5028f3a4e6db005afe25845"} Oct 30 22:00:10.748426 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.748113 4560 scope.go:122] "RemoveContainer" containerID="9e97a8c9bbf328276d7147f509d20b591fa5833094c6d49562c80de24c9ea5db" Oct 30 22:00:10.748426 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.748292 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-40bb78-6d7d4d7644-jwwbv" Oct 30 22:00:10.772962 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.769643 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f4405ac-93f4-4a1d-a335-5990238c4294-istio-envoy\") pod \"e2e-api-a-28c192-openshift-default-5586f598c-r8qq9\" (UID: \"7f4405ac-93f4-4a1d-a335-5990238c4294\") " pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" Oct 30 22:00:10.776916 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.776886 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f4405ac-93f4-4a1d-a335-5990238c4294-istio-podinfo\") pod \"e2e-api-a-28c192-openshift-default-5586f598c-r8qq9\" (UID: \"7f4405ac-93f4-4a1d-a335-5990238c4294\") " pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" Oct 30 22:00:10.779367 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.779091 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f4405ac-93f4-4a1d-a335-5990238c4294-istio-token\") pod \"e2e-api-a-28c192-openshift-default-5586f598c-r8qq9\" (UID: \"7f4405ac-93f4-4a1d-a335-5990238c4294\") " pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" Oct 30 22:00:10.779754 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.779731 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jvln\" (UniqueName: \"kubernetes.io/projected/7f4405ac-93f4-4a1d-a335-5990238c4294-kube-api-access-7jvln\") pod \"e2e-api-a-28c192-openshift-default-5586f598c-r8qq9\" (UID: \"7f4405ac-93f4-4a1d-a335-5990238c4294\") " pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" Oct 30 22:00:10.814907 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.814629 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" Oct 30 22:00:10.862388 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.856442 4560 patch_prober.go:29] interesting pod/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 22:00:10.862388 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.856496 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" podUID="156f466e-16e0-441c-9ce5-796acd8551b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 22:00:10.917623 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.917587 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-40bb78-6d7d4d7644-jwwbv"] Oct 30 22:00:10.919855 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:10.919822 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-40bb78-6d7d4d7644-jwwbv"] Oct 30 22:00:11.363154 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:11.363038 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9"] Oct 30 22:00:11.365808 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:00:11.365781 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f4405ac_93f4_4a1d_a335_5990238c4294.slice/crio-cb20ac229596075cb5e8ee9ad4ca8d42fc90f6c4b723073a435e179e830959e2 WatchSource:0}: Error finding container cb20ac229596075cb5e8ee9ad4ca8d42fc90f6c4b723073a435e179e830959e2: Status 404 returned error can't find the container with id cb20ac229596075cb5e8ee9ad4ca8d42fc90f6c4b723073a435e179e830959e2 Oct 30 22:00:11.369166 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:11.369126 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:00:11.369260 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:11.369213 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:00:11.369310 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:11.369257 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:00:11.766106 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:11.766068 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" event={"ID":"7f4405ac-93f4-4a1d-a335-5990238c4294","Type":"ContainerStarted","Data":"de6e083df99964d0fec1de18f4a318649336521e9d7ad8b20d61d0c3d1dc1c10"} Oct 30 22:00:11.766296 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:11.766130 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" event={"ID":"7f4405ac-93f4-4a1d-a335-5990238c4294","Type":"ContainerStarted","Data":"cb20ac229596075cb5e8ee9ad4ca8d42fc90f6c4b723073a435e179e830959e2"} Oct 30 22:00:11.815280 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:11.815250 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" Oct 30 22:00:11.857729 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:11.857687 4560 patch_prober.go:29] interesting pod/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 22:00:11.857942 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:11.857752 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" podUID="156f466e-16e0-441c-9ce5-796acd8551b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 22:00:11.888479 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:11.888301 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" podStartSLOduration=1.888282399 podStartE2EDuration="1.888282399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:11.883650479 +0000 UTC m=+2787.438350257" watchObservedRunningTime="2026-10-30 22:00:11.888282399 +0000 UTC m=+2787.442982176" Oct 30 22:00:12.587193 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:12.587154 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="345a9ec6-062e-4402-83fc-15adac6800bd" path="/var/lib/kubelet/pods/345a9ec6-062e-4402-83fc-15adac6800bd/volumes" Oct 30 22:00:12.821571 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:12.821534 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" Oct 30 22:00:12.977811 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:12.977775 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" Oct 30 22:00:12.978251 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:12.978232 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" Oct 30 22:00:12.979419 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:12.979400 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" Oct 30 22:00:13.780514 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:13.780484 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" Oct 30 22:00:13.781367 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:13.781328 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" Oct 30 22:00:14.572930 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:14.572889 4560 scope.go:122] "RemoveContainer" containerID="00f4fb6ff34433cfd34875df31fdca4f236c405ab95752939c3555ba243010b6" Oct 30 22:00:15.569057 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:15.569033 4560 scope.go:122] "RemoveContainer" containerID="75bc772204d89e6085ef8af9a5a5936ea3951f0f2623987ef15fb4ca27850daf" Oct 30 22:00:15.569501 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:15.569179 4560 scope.go:122] "RemoveContainer" containerID="e87c29eacbd59127c77ea19b5e354c90b868c2d59b49741f1b36a19dc991bc87" Oct 30 22:00:15.569501 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:15.569277 4560 scope.go:122] "RemoveContainer" containerID="c24a3ceba42c926ab2a75bc1a1fb8d1d5c740c87d5b71335f0bd11b2a4f75ec3" Oct 30 22:00:15.796209 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:15.796179 4560 generic.go:356] "Generic (PLEG): container finished" podID="d29a3644-c4e8-40eb-a34b-f531c2a1565b" containerID="012e5edd0a1e0eec0244aa11254ab60b2ab2fe291462f7a76531ab4429e74d5d" exitCode=60 Oct 30 22:00:15.796335 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:15.796217 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ed6135-29889960-bfnmd" event={"ID":"d29a3644-c4e8-40eb-a34b-f531c2a1565b","Type":"ContainerDied","Data":"012e5edd0a1e0eec0244aa11254ab60b2ab2fe291462f7a76531ab4429e74d5d"} Oct 30 22:00:15.796335 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:15.796282 4560 scope.go:122] "RemoveContainer" containerID="00f4fb6ff34433cfd34875df31fdca4f236c405ab95752939c3555ba243010b6" Oct 30 22:00:15.796674 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:15.796654 4560 scope.go:122] "RemoveContainer" containerID="012e5edd0a1e0eec0244aa11254ab60b2ab2fe291462f7a76531ab4429e74d5d" Oct 30 22:00:15.796967 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:00:15.796939 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-ed6135-29889960-bfnmd_odh-ai-gateway-infra(d29a3644-c4e8-40eb-a34b-f531c2a1565b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ed6135-29889960-bfnmd" podUID="d29a3644-c4e8-40eb-a34b-f531c2a1565b" Oct 30 22:00:16.569468 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:16.569442 4560 scope.go:122] "RemoveContainer" containerID="638b6de6d5467077ed58a6a006c3f5ac2225c5e8f045711425d695058ae56210" Oct 30 22:00:16.569933 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:16.569906 4560 scope.go:122] "RemoveContainer" containerID="517cf60d1f265380c91e0f76fe2e077f2346d19ef39b910e52ccf1c356472882" Oct 30 22:00:16.804149 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:16.804103 4560 generic.go:356] "Generic (PLEG): container finished" podID="08796630-f181-4b69-bd3c-32f627dff3ac" containerID="553a58f922a9d0442d8d96b40cce0514d878029d73f6960ef2b33bbf5835bd70" exitCode=60 Oct 30 22:00:16.804374 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:16.804177 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2df7b6-29889960-96qhk" event={"ID":"08796630-f181-4b69-bd3c-32f627dff3ac","Type":"ContainerDied","Data":"553a58f922a9d0442d8d96b40cce0514d878029d73f6960ef2b33bbf5835bd70"} Oct 30 22:00:16.804374 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:16.804235 4560 scope.go:122] "RemoveContainer" containerID="75bc772204d89e6085ef8af9a5a5936ea3951f0f2623987ef15fb4ca27850daf" Oct 30 22:00:16.804694 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:16.804668 4560 scope.go:122] "RemoveContainer" containerID="553a58f922a9d0442d8d96b40cce0514d878029d73f6960ef2b33bbf5835bd70" Oct 30 22:00:16.805015 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:00:16.804989 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-2df7b6-29889960-96qhk_odh-ai-gateway-infra(08796630-f181-4b69-bd3c-32f627dff3ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2df7b6-29889960-96qhk" podUID="08796630-f181-4b69-bd3c-32f627dff3ac" Oct 30 22:00:16.807594 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:16.807574 4560 generic.go:356] "Generic (PLEG): container finished" podID="a24ad108-152e-4690-82ae-71e98f901c61" containerID="c90e4ae270111d37612b9dac42c28de3afb4e93eabce0a015210a27c1f00920d" exitCode=60 Oct 30 22:00:16.807704 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:16.807652 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-40bb78-29889960-98vl4" event={"ID":"a24ad108-152e-4690-82ae-71e98f901c61","Type":"ContainerDied","Data":"c90e4ae270111d37612b9dac42c28de3afb4e93eabce0a015210a27c1f00920d"} Oct 30 22:00:16.808006 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:16.807988 4560 scope.go:122] "RemoveContainer" containerID="c90e4ae270111d37612b9dac42c28de3afb4e93eabce0a015210a27c1f00920d" Oct 30 22:00:16.808299 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:00:16.808270 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-40bb78-29889960-98vl4_odh-ai-gateway-infra(a24ad108-152e-4690-82ae-71e98f901c61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-40bb78-29889960-98vl4" podUID="a24ad108-152e-4690-82ae-71e98f901c61" Oct 30 22:00:16.809495 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:16.809475 4560 generic.go:356] "Generic (PLEG): container finished" podID="ae04a1f9-8997-42e2-b7e9-94e84fd97d71" containerID="167af646fde9fef30a988f49e4065c1048e84f30f8efc3f89192d08eca49ad5d" exitCode=60 Oct 30 22:00:16.809579 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:16.809539 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-bzl5x" event={"ID":"ae04a1f9-8997-42e2-b7e9-94e84fd97d71","Type":"ContainerDied","Data":"167af646fde9fef30a988f49e4065c1048e84f30f8efc3f89192d08eca49ad5d"} Oct 30 22:00:16.809799 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:16.809786 4560 scope.go:122] "RemoveContainer" containerID="167af646fde9fef30a988f49e4065c1048e84f30f8efc3f89192d08eca49ad5d" Oct 30 22:00:16.809960 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:00:16.809945 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889960-bzl5x_odh-ai-gateway-infra(ae04a1f9-8997-42e2-b7e9-94e84fd97d71)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-bzl5x" podUID="ae04a1f9-8997-42e2-b7e9-94e84fd97d71" Oct 30 22:00:16.818114 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:16.818089 4560 scope.go:122] "RemoveContainer" containerID="c24a3ceba42c926ab2a75bc1a1fb8d1d5c740c87d5b71335f0bd11b2a4f75ec3" Oct 30 22:00:16.831609 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:16.831588 4560 scope.go:122] "RemoveContainer" containerID="e87c29eacbd59127c77ea19b5e354c90b868c2d59b49741f1b36a19dc991bc87" Oct 30 22:00:16.928316 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:16.928275 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ed6135-29889960-bfnmd"] Oct 30 22:00:17.067581 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:17.067532 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ed6135-29889960-bfnmd" Oct 30 22:00:17.100477 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:17.100452 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d29a3644-c4e8-40eb-a34b-f531c2a1565b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d29a3644-c4e8-40eb-a34b-f531c2a1565b-maas-api-service-ca\") pod \"d29a3644-c4e8-40eb-a34b-f531c2a1565b\" (UID: \"d29a3644-c4e8-40eb-a34b-f531c2a1565b\") " Oct 30 22:00:17.100590 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:17.100494 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d29a3644-c4e8-40eb-a34b-f531c2a1565b-kube-api-access-n6h7s\" (UniqueName: \"kubernetes.io/projected/d29a3644-c4e8-40eb-a34b-f531c2a1565b-kube-api-access-n6h7s\") pod \"d29a3644-c4e8-40eb-a34b-f531c2a1565b\" (UID: \"d29a3644-c4e8-40eb-a34b-f531c2a1565b\") " Oct 30 22:00:17.100758 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:17.100738 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d29a3644-c4e8-40eb-a34b-f531c2a1565b-maas-api-service-ca" pod "d29a3644-c4e8-40eb-a34b-f531c2a1565b" (UID: "d29a3644-c4e8-40eb-a34b-f531c2a1565b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:17.102559 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:17.102539 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d29a3644-c4e8-40eb-a34b-f531c2a1565b-kube-api-access-n6h7s" pod "d29a3644-c4e8-40eb-a34b-f531c2a1565b" (UID: "d29a3644-c4e8-40eb-a34b-f531c2a1565b"). InnerVolumeSpecName "kube-api-access-n6h7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:17.202175 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:17.202141 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d29a3644-c4e8-40eb-a34b-f531c2a1565b-maas-api-service-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:17.202175 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:17.202177 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6h7s\" (UniqueName: \"kubernetes.io/projected/d29a3644-c4e8-40eb-a34b-f531c2a1565b-kube-api-access-n6h7s\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:17.569127 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:17.569102 4560 scope.go:122] "RemoveContainer" containerID="21e5282fcca578aa6c4ac1207c6d6ee96e54dfb9405886ce607d46c70169119a" Oct 30 22:00:17.569276 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:17.569211 4560 scope.go:122] "RemoveContainer" containerID="f77a5a08563c00cb4cb2825d1590f04fafa9b23c5b69492e0ddf8f2f916f22a1" Oct 30 22:00:17.818789 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:17.818706 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ed6135-29889960-bfnmd" Oct 30 22:00:17.818789 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:17.818710 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ed6135-29889960-bfnmd" event={"ID":"d29a3644-c4e8-40eb-a34b-f531c2a1565b","Type":"ContainerDied","Data":"8738dc6c6fca3c51faf1d9f91c9aaed08dc307540a3506aea34bcbffd86a97eb"} Oct 30 22:00:17.819275 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:17.818833 4560 scope.go:122] "RemoveContainer" containerID="012e5edd0a1e0eec0244aa11254ab60b2ab2fe291462f7a76531ab4429e74d5d" Oct 30 22:00:17.820743 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:17.820722 4560 generic.go:356] "Generic (PLEG): container finished" podID="e174ecd3-9e50-458b-85aa-b130ceba6d70" containerID="f41ff89bb594d0293ab8cc97da6b8db6e0a8838c106e6345c5d4ad7717738b2f" exitCode=60 Oct 30 22:00:17.820866 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:17.820791 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889960-vqpcn" event={"ID":"e174ecd3-9e50-458b-85aa-b130ceba6d70","Type":"ContainerDied","Data":"f41ff89bb594d0293ab8cc97da6b8db6e0a8838c106e6345c5d4ad7717738b2f"} Oct 30 22:00:17.821187 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:17.821165 4560 scope.go:122] "RemoveContainer" containerID="f41ff89bb594d0293ab8cc97da6b8db6e0a8838c106e6345c5d4ad7717738b2f" Oct 30 22:00:17.821454 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:00:17.821434 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889960-vqpcn_odh-ai-gateway-infra(e174ecd3-9e50-458b-85aa-b130ceba6d70)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889960-vqpcn" podUID="e174ecd3-9e50-458b-85aa-b130ceba6d70" Oct 30 22:00:17.824703 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:17.824683 4560 generic.go:356] "Generic (PLEG): container finished" podID="aa5fbac8-0bd8-4793-bbaa-c8dae197566d" containerID="9bc946e232e234ca296f20eb9361a7c0b0117fb7d9faab67bc62e535d87fa805" exitCode=60 Oct 30 22:00:17.824802 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:17.824752 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19dd4f-29889960-hcm42" event={"ID":"aa5fbac8-0bd8-4793-bbaa-c8dae197566d","Type":"ContainerDied","Data":"9bc946e232e234ca296f20eb9361a7c0b0117fb7d9faab67bc62e535d87fa805"} Oct 30 22:00:17.825107 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:17.825087 4560 scope.go:122] "RemoveContainer" containerID="9bc946e232e234ca296f20eb9361a7c0b0117fb7d9faab67bc62e535d87fa805" Oct 30 22:00:17.825333 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:00:17.825317 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-19dd4f-29889960-hcm42_odh-ai-gateway-infra(aa5fbac8-0bd8-4793-bbaa-c8dae197566d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19dd4f-29889960-hcm42" podUID="aa5fbac8-0bd8-4793-bbaa-c8dae197566d" Oct 30 22:00:17.829695 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:17.829680 4560 scope.go:122] "RemoveContainer" containerID="638b6de6d5467077ed58a6a006c3f5ac2225c5e8f045711425d695058ae56210" Oct 30 22:00:17.846316 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:17.846298 4560 scope.go:122] "RemoveContainer" containerID="517cf60d1f265380c91e0f76fe2e077f2346d19ef39b910e52ccf1c356472882" Oct 30 22:00:17.934269 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:17.934243 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2df7b6-29889960-96qhk"] Oct 30 22:00:18.061291 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.061257 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-40bb78-29889960-98vl4"] Oct 30 22:00:18.097582 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.097552 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ed6135-29889960-bfnmd"] Oct 30 22:00:18.103317 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.103282 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ed6135-29889960-bfnmd"] Oct 30 22:00:18.111556 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.111535 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2df7b6-29889960-96qhk" Oct 30 22:00:18.226811 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.226769 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08796630-f181-4b69-bd3c-32f627dff3ac-kube-api-access-ptw5z\" (UniqueName: \"kubernetes.io/projected/08796630-f181-4b69-bd3c-32f627dff3ac-kube-api-access-ptw5z\") pod \"08796630-f181-4b69-bd3c-32f627dff3ac\" (UID: \"08796630-f181-4b69-bd3c-32f627dff3ac\") " Oct 30 22:00:18.227223 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.226966 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08796630-f181-4b69-bd3c-32f627dff3ac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08796630-f181-4b69-bd3c-32f627dff3ac-maas-api-service-ca\") pod \"08796630-f181-4b69-bd3c-32f627dff3ac\" (UID: \"08796630-f181-4b69-bd3c-32f627dff3ac\") " Oct 30 22:00:18.228396 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.228323 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08796630-f181-4b69-bd3c-32f627dff3ac-maas-api-service-ca" pod "08796630-f181-4b69-bd3c-32f627dff3ac" (UID: "08796630-f181-4b69-bd3c-32f627dff3ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:18.263519 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.263484 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-bzl5x"] Oct 30 22:00:18.271953 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.271906 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08796630-f181-4b69-bd3c-32f627dff3ac-kube-api-access-ptw5z" pod "08796630-f181-4b69-bd3c-32f627dff3ac" (UID: "08796630-f181-4b69-bd3c-32f627dff3ac"). InnerVolumeSpecName "kube-api-access-ptw5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:18.318832 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.318766 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-40bb78-29889960-98vl4" Oct 30 22:00:18.328785 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.328761 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a24ad108-152e-4690-82ae-71e98f901c61-kube-api-access-n6ll8\" (UniqueName: \"kubernetes.io/projected/a24ad108-152e-4690-82ae-71e98f901c61-kube-api-access-n6ll8\") pod \"a24ad108-152e-4690-82ae-71e98f901c61\" (UID: \"a24ad108-152e-4690-82ae-71e98f901c61\") " Oct 30 22:00:18.328888 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.328867 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a24ad108-152e-4690-82ae-71e98f901c61-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a24ad108-152e-4690-82ae-71e98f901c61-maas-api-service-ca\") pod \"a24ad108-152e-4690-82ae-71e98f901c61\" (UID: \"a24ad108-152e-4690-82ae-71e98f901c61\") " Oct 30 22:00:18.329399 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.329368 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a24ad108-152e-4690-82ae-71e98f901c61-maas-api-service-ca" pod "a24ad108-152e-4690-82ae-71e98f901c61" (UID: "a24ad108-152e-4690-82ae-71e98f901c61"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:18.329499 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.329433 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptw5z\" (UniqueName: \"kubernetes.io/projected/08796630-f181-4b69-bd3c-32f627dff3ac-kube-api-access-ptw5z\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:18.329499 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.329456 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08796630-f181-4b69-bd3c-32f627dff3ac-maas-api-service-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:18.331901 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.331792 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a24ad108-152e-4690-82ae-71e98f901c61-kube-api-access-n6ll8" pod "a24ad108-152e-4690-82ae-71e98f901c61" (UID: "a24ad108-152e-4690-82ae-71e98f901c61"). InnerVolumeSpecName "kube-api-access-n6ll8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:18.443762 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.433458 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a24ad108-152e-4690-82ae-71e98f901c61-maas-api-service-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:18.443762 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.433586 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6ll8\" (UniqueName: \"kubernetes.io/projected/a24ad108-152e-4690-82ae-71e98f901c61-kube-api-access-n6ll8\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:18.476775 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.476746 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-bzl5x" Oct 30 22:00:18.534873 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.534838 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae04a1f9-8997-42e2-b7e9-94e84fd97d71-kube-api-access-bfn98\" (UniqueName: \"kubernetes.io/projected/ae04a1f9-8997-42e2-b7e9-94e84fd97d71-kube-api-access-bfn98\") pod \"ae04a1f9-8997-42e2-b7e9-94e84fd97d71\" (UID: \"ae04a1f9-8997-42e2-b7e9-94e84fd97d71\") " Oct 30 22:00:18.535067 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.535049 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae04a1f9-8997-42e2-b7e9-94e84fd97d71-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae04a1f9-8997-42e2-b7e9-94e84fd97d71-maas-api-service-ca\") pod \"ae04a1f9-8997-42e2-b7e9-94e84fd97d71\" (UID: \"ae04a1f9-8997-42e2-b7e9-94e84fd97d71\") " Oct 30 22:00:18.535545 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.535509 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae04a1f9-8997-42e2-b7e9-94e84fd97d71-maas-api-service-ca" pod "ae04a1f9-8997-42e2-b7e9-94e84fd97d71" (UID: "ae04a1f9-8997-42e2-b7e9-94e84fd97d71"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:18.536910 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.536886 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae04a1f9-8997-42e2-b7e9-94e84fd97d71-kube-api-access-bfn98" pod "ae04a1f9-8997-42e2-b7e9-94e84fd97d71" (UID: "ae04a1f9-8997-42e2-b7e9-94e84fd97d71"). InnerVolumeSpecName "kube-api-access-bfn98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:18.569406 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.569373 4560 scope.go:122] "RemoveContainer" containerID="7e35cba6c2c2eefb871732dc86019ac43af6a8dff90923cd4959be7bbee1ed2e" Oct 30 22:00:18.569616 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.569601 4560 scope.go:122] "RemoveContainer" containerID="530c1b6ba227741dcd0858722dbd45d78df6331d8e2d85276ace8307da4653e2" Oct 30 22:00:18.577207 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.577184 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d29a3644-c4e8-40eb-a34b-f531c2a1565b" path="/var/lib/kubelet/pods/d29a3644-c4e8-40eb-a34b-f531c2a1565b/volumes" Oct 30 22:00:18.658056 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.637181 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae04a1f9-8997-42e2-b7e9-94e84fd97d71-maas-api-service-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:18.658056 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.637234 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfn98\" (UniqueName: \"kubernetes.io/projected/ae04a1f9-8997-42e2-b7e9-94e84fd97d71-kube-api-access-bfn98\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:18.838010 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.837936 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-40bb78-29889960-98vl4" Oct 30 22:00:18.838455 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.837937 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-40bb78-29889960-98vl4" event={"ID":"a24ad108-152e-4690-82ae-71e98f901c61","Type":"ContainerDied","Data":"4431c088644cbb2d50d13ad1a1d22156c486ddfcd4cf74196adddeacdc74e2aa"} Oct 30 22:00:18.838455 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.838070 4560 scope.go:122] "RemoveContainer" containerID="c90e4ae270111d37612b9dac42c28de3afb4e93eabce0a015210a27c1f00920d" Oct 30 22:00:18.840003 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.839982 4560 generic.go:356] "Generic (PLEG): container finished" podID="818309d7-73a4-404f-9942-5bcbb2a4fd14" containerID="3cd924f24af4d7b736dd657432335eb960a31f73ee86b9d2356758e6b76333c4" exitCode=60 Oct 30 22:00:18.840126 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.840052 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889960-tfsx2" event={"ID":"818309d7-73a4-404f-9942-5bcbb2a4fd14","Type":"ContainerDied","Data":"3cd924f24af4d7b736dd657432335eb960a31f73ee86b9d2356758e6b76333c4"} Oct 30 22:00:18.840496 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.840476 4560 scope.go:122] "RemoveContainer" containerID="3cd924f24af4d7b736dd657432335eb960a31f73ee86b9d2356758e6b76333c4" Oct 30 22:00:18.840847 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:00:18.840824 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-554f46-29889960-tfsx2_odh-ai-gateway-infra(818309d7-73a4-404f-9942-5bcbb2a4fd14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889960-tfsx2" podUID="818309d7-73a4-404f-9942-5bcbb2a4fd14" Oct 30 22:00:18.842966 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.842949 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-bzl5x" Oct 30 22:00:18.843044 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.842984 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-bzl5x" event={"ID":"ae04a1f9-8997-42e2-b7e9-94e84fd97d71","Type":"ContainerDied","Data":"3b00a2f4eef5eef01c3cf727040ca9147ec1e6750a2441c6aea337f1aa0741d4"} Oct 30 22:00:18.844671 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.844651 4560 generic.go:356] "Generic (PLEG): container finished" podID="45e6d264-3503-493e-95f2-6308e04e93ab" containerID="eef10b579a71d82c82297491c17065d1a714b5a122f7e786e33d011cf30f0815" exitCode=60 Oct 30 22:00:18.844749 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.844717 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889960-g9zpz" event={"ID":"45e6d264-3503-493e-95f2-6308e04e93ab","Type":"ContainerDied","Data":"eef10b579a71d82c82297491c17065d1a714b5a122f7e786e33d011cf30f0815"} Oct 30 22:00:18.845095 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.845071 4560 scope.go:122] "RemoveContainer" containerID="eef10b579a71d82c82297491c17065d1a714b5a122f7e786e33d011cf30f0815" Oct 30 22:00:18.845316 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:00:18.845299 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-71aab5-29889960-g9zpz_odh-ai-gateway-infra(45e6d264-3503-493e-95f2-6308e04e93ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889960-g9zpz" podUID="45e6d264-3503-493e-95f2-6308e04e93ab" Oct 30 22:00:18.845928 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.845913 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2df7b6-29889960-96qhk" Oct 30 22:00:18.845997 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.845913 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2df7b6-29889960-96qhk" event={"ID":"08796630-f181-4b69-bd3c-32f627dff3ac","Type":"ContainerDied","Data":"567b7d39a2ae638f9da801bfd351d496cebe212be001f4b67936d05512e58b69"} Oct 30 22:00:18.858521 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.858506 4560 scope.go:122] "RemoveContainer" containerID="f77a5a08563c00cb4cb2825d1590f04fafa9b23c5b69492e0ddf8f2f916f22a1" Oct 30 22:00:18.874612 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.874596 4560 scope.go:122] "RemoveContainer" containerID="167af646fde9fef30a988f49e4065c1048e84f30f8efc3f89192d08eca49ad5d" Oct 30 22:00:18.882797 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.882778 4560 scope.go:122] "RemoveContainer" containerID="21e5282fcca578aa6c4ac1207c6d6ee96e54dfb9405886ce607d46c70169119a" Oct 30 22:00:18.901571 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:18.900329 4560 scope.go:122] "RemoveContainer" containerID="553a58f922a9d0442d8d96b40cce0514d878029d73f6960ef2b33bbf5835bd70" Oct 30 22:00:19.007481 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:19.007453 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889960-vqpcn"] Oct 30 22:00:19.010812 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:19.010792 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2df7b6-29889960-96qhk"] Oct 30 22:00:19.018092 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:19.018071 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2df7b6-29889960-96qhk"] Oct 30 22:00:19.133675 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:19.133587 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-bzl5x"] Oct 30 22:00:19.145792 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:19.145767 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-bzl5x"] Oct 30 22:00:19.166871 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:19.166846 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889960-vqpcn" Oct 30 22:00:19.241549 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:19.241520 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19dd4f-29889960-hcm42"] Oct 30 22:00:19.262955 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:19.262921 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-40bb78-29889960-98vl4"] Oct 30 22:00:19.267368 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:19.266453 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-40bb78-29889960-98vl4"] Oct 30 22:00:19.267368 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:19.266813 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e174ecd3-9e50-458b-85aa-b130ceba6d70-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e174ecd3-9e50-458b-85aa-b130ceba6d70-maas-api-service-ca\") pod \"e174ecd3-9e50-458b-85aa-b130ceba6d70\" (UID: \"e174ecd3-9e50-458b-85aa-b130ceba6d70\") " Oct 30 22:00:19.267368 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:19.267095 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e174ecd3-9e50-458b-85aa-b130ceba6d70-kube-api-access-2wbh2\" (UniqueName: \"kubernetes.io/projected/e174ecd3-9e50-458b-85aa-b130ceba6d70-kube-api-access-2wbh2\") pod \"e174ecd3-9e50-458b-85aa-b130ceba6d70\" (UID: \"e174ecd3-9e50-458b-85aa-b130ceba6d70\") " Oct 30 22:00:19.267368 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:19.267119 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e174ecd3-9e50-458b-85aa-b130ceba6d70-maas-api-service-ca" pod "e174ecd3-9e50-458b-85aa-b130ceba6d70" (UID: "e174ecd3-9e50-458b-85aa-b130ceba6d70"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:19.267723 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:19.267521 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e174ecd3-9e50-458b-85aa-b130ceba6d70-maas-api-service-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:19.271390 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:19.269202 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e174ecd3-9e50-458b-85aa-b130ceba6d70-kube-api-access-2wbh2" pod "e174ecd3-9e50-458b-85aa-b130ceba6d70" (UID: "e174ecd3-9e50-458b-85aa-b130ceba6d70"). InnerVolumeSpecName "kube-api-access-2wbh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:19.362160 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:19.362135 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19dd4f-29889960-hcm42" Oct 30 22:00:19.368613 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:19.368592 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wbh2\" (UniqueName: \"kubernetes.io/projected/e174ecd3-9e50-458b-85aa-b130ceba6d70-kube-api-access-2wbh2\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:19.469641 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:19.469555 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa5fbac8-0bd8-4793-bbaa-c8dae197566d-kube-api-access-nx2fc\" (UniqueName: \"kubernetes.io/projected/aa5fbac8-0bd8-4793-bbaa-c8dae197566d-kube-api-access-nx2fc\") pod \"aa5fbac8-0bd8-4793-bbaa-c8dae197566d\" (UID: \"aa5fbac8-0bd8-4793-bbaa-c8dae197566d\") " Oct 30 22:00:19.469779 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:19.469645 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa5fbac8-0bd8-4793-bbaa-c8dae197566d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa5fbac8-0bd8-4793-bbaa-c8dae197566d-maas-api-service-ca\") pod \"aa5fbac8-0bd8-4793-bbaa-c8dae197566d\" (UID: \"aa5fbac8-0bd8-4793-bbaa-c8dae197566d\") " Oct 30 22:00:19.469988 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:19.469964 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa5fbac8-0bd8-4793-bbaa-c8dae197566d-maas-api-service-ca" pod "aa5fbac8-0bd8-4793-bbaa-c8dae197566d" (UID: "aa5fbac8-0bd8-4793-bbaa-c8dae197566d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:19.471705 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:19.471679 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa5fbac8-0bd8-4793-bbaa-c8dae197566d-kube-api-access-nx2fc" pod "aa5fbac8-0bd8-4793-bbaa-c8dae197566d" (UID: "aa5fbac8-0bd8-4793-bbaa-c8dae197566d"). InnerVolumeSpecName "kube-api-access-nx2fc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:19.570668 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:19.570637 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx2fc\" (UniqueName: \"kubernetes.io/projected/aa5fbac8-0bd8-4793-bbaa-c8dae197566d-kube-api-access-nx2fc\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:19.570668 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:19.570664 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa5fbac8-0bd8-4793-bbaa-c8dae197566d-maas-api-service-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:19.686281 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:00:19.686242 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:00:19.857266 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:19.857231 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889960-vqpcn" Oct 30 22:00:19.857764 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:19.857272 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889960-vqpcn" event={"ID":"e174ecd3-9e50-458b-85aa-b130ceba6d70","Type":"ContainerDied","Data":"f2f202cca41bbb2b95701b97e27307a9d61d604683010afc06138fc38a0888e8"} Oct 30 22:00:19.857764 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:19.857366 4560 scope.go:122] "RemoveContainer" containerID="f41ff89bb594d0293ab8cc97da6b8db6e0a8838c106e6345c5d4ad7717738b2f" Oct 30 22:00:19.861844 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:19.861824 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19dd4f-29889960-hcm42" Oct 30 22:00:19.861950 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:19.861828 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19dd4f-29889960-hcm42" event={"ID":"aa5fbac8-0bd8-4793-bbaa-c8dae197566d","Type":"ContainerDied","Data":"1508e941687c973b8b3b6f3df8d3c83c0e99799842abf263d6d41278e09c012b"} Oct 30 22:00:19.864862 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:19.864837 4560 generic.go:356] "Generic (PLEG): container finished" podID="d8a90e7a-b379-4742-a167-01aabd410bf9" containerID="b440b6c4d46b6acecc67671f6fba1a1433ccccfc119b90e6416ebca602a8dd69" exitCode=60 Oct 30 22:00:19.864978 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:19.864907 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ca0a55-29889960-8wsx6" event={"ID":"d8a90e7a-b379-4742-a167-01aabd410bf9","Type":"ContainerDied","Data":"b440b6c4d46b6acecc67671f6fba1a1433ccccfc119b90e6416ebca602a8dd69"} Oct 30 22:00:19.865350 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:19.865315 4560 scope.go:122] "RemoveContainer" containerID="b440b6c4d46b6acecc67671f6fba1a1433ccccfc119b90e6416ebca602a8dd69" Oct 30 22:00:19.865646 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:00:19.865621 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-ca0a55-29889960-8wsx6_odh-ai-gateway-infra(d8a90e7a-b379-4742-a167-01aabd410bf9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ca0a55-29889960-8wsx6" podUID="d8a90e7a-b379-4742-a167-01aabd410bf9" Oct 30 22:00:19.866979 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:19.866960 4560 generic.go:356] "Generic (PLEG): container finished" podID="22606928-4bc3-49e5-91e9-00497b022b10" containerID="117d50ae00eccab7b8fe9e45b10f09767775de6a8ad9841886c2c006ece320d8" exitCode=60 Oct 30 22:00:19.867067 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:19.866981 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bfdf30b9-29889960-4c9wb" event={"ID":"22606928-4bc3-49e5-91e9-00497b022b10","Type":"ContainerDied","Data":"117d50ae00eccab7b8fe9e45b10f09767775de6a8ad9841886c2c006ece320d8"} Oct 30 22:00:19.867555 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:19.867532 4560 scope.go:122] "RemoveContainer" containerID="117d50ae00eccab7b8fe9e45b10f09767775de6a8ad9841886c2c006ece320d8" Oct 30 22:00:19.867902 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:00:19.867789 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-bfdf30b9-29889960-4c9wb_odh-ai-gateway-infra(22606928-4bc3-49e5-91e9-00497b022b10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bfdf30b9-29889960-4c9wb" podUID="22606928-4bc3-49e5-91e9-00497b022b10" Oct 30 22:00:19.869167 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:19.869150 4560 scope.go:122] "RemoveContainer" containerID="9bc946e232e234ca296f20eb9361a7c0b0117fb7d9faab67bc62e535d87fa805" Oct 30 22:00:19.877701 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:19.877684 4560 scope.go:122] "RemoveContainer" containerID="530c1b6ba227741dcd0858722dbd45d78df6331d8e2d85276ace8307da4653e2" Oct 30 22:00:19.889462 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:19.889445 4560 scope.go:122] "RemoveContainer" containerID="7e35cba6c2c2eefb871732dc86019ac43af6a8dff90923cd4959be7bbee1ed2e" Oct 30 22:00:20.015880 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.015849 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889960-vqpcn"] Oct 30 22:00:20.026042 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.026008 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-afbe5b-29889960-vqpcn"] Oct 30 22:00:20.255179 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.255148 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19dd4f-29889960-hcm42"] Oct 30 22:00:20.278358 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.260398 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19dd4f-29889960-hcm42"] Oct 30 22:00:20.397109 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.397079 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889960-g9zpz"] Oct 30 22:00:20.520455 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.520361 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889960-tfsx2"] Oct 30 22:00:20.552086 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.552057 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889960-g9zpz" Oct 30 22:00:20.577864 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.577835 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08796630-f181-4b69-bd3c-32f627dff3ac" path="/var/lib/kubelet/pods/08796630-f181-4b69-bd3c-32f627dff3ac/volumes" Oct 30 22:00:20.578362 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.578318 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a24ad108-152e-4690-82ae-71e98f901c61" path="/var/lib/kubelet/pods/a24ad108-152e-4690-82ae-71e98f901c61/volumes" Oct 30 22:00:20.578877 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.578851 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa5fbac8-0bd8-4793-bbaa-c8dae197566d" path="/var/lib/kubelet/pods/aa5fbac8-0bd8-4793-bbaa-c8dae197566d/volumes" Oct 30 22:00:20.579794 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.579776 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae04a1f9-8997-42e2-b7e9-94e84fd97d71" path="/var/lib/kubelet/pods/ae04a1f9-8997-42e2-b7e9-94e84fd97d71/volumes" Oct 30 22:00:20.580195 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.580182 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e174ecd3-9e50-458b-85aa-b130ceba6d70" path="/var/lib/kubelet/pods/e174ecd3-9e50-458b-85aa-b130ceba6d70/volumes" Oct 30 22:00:20.650122 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.650092 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p"] Oct 30 22:00:20.650837 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.650814 4560 cpu_manager.go:414] "containerMap: removing container" podUID="45e6d264-3503-493e-95f2-6308e04e93ab" containerName="cleanup" Oct 30 22:00:20.650837 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.650835 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45e6d264-3503-493e-95f2-6308e04e93ab" containerName="cleanup" Oct 30 22:00:20.651092 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.650854 4560 cpu_manager.go:414] "containerMap: removing container" podUID="a24ad108-152e-4690-82ae-71e98f901c61" containerName="cleanup" Oct 30 22:00:20.651092 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.650862 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a24ad108-152e-4690-82ae-71e98f901c61" containerName="cleanup" Oct 30 22:00:20.651092 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.650873 4560 cpu_manager.go:414] "containerMap: removing container" podUID="ae04a1f9-8997-42e2-b7e9-94e84fd97d71" containerName="cleanup" Oct 30 22:00:20.651092 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.650880 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae04a1f9-8997-42e2-b7e9-94e84fd97d71" containerName="cleanup" Oct 30 22:00:20.651092 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.650893 4560 cpu_manager.go:414] "containerMap: removing container" podUID="d29a3644-c4e8-40eb-a34b-f531c2a1565b" containerName="cleanup" Oct 30 22:00:20.651092 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.650899 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d29a3644-c4e8-40eb-a34b-f531c2a1565b" containerName="cleanup" Oct 30 22:00:20.651092 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.650906 4560 cpu_manager.go:414] "containerMap: removing container" podUID="08796630-f181-4b69-bd3c-32f627dff3ac" containerName="cleanup" Oct 30 22:00:20.651092 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.650911 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08796630-f181-4b69-bd3c-32f627dff3ac" containerName="cleanup" Oct 30 22:00:20.651092 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.650922 4560 cpu_manager.go:414] "containerMap: removing container" podUID="e174ecd3-9e50-458b-85aa-b130ceba6d70" containerName="cleanup" Oct 30 22:00:20.651092 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.650926 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e174ecd3-9e50-458b-85aa-b130ceba6d70" containerName="cleanup" Oct 30 22:00:20.651092 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.650949 4560 cpu_manager.go:414] "containerMap: removing container" podUID="aa5fbac8-0bd8-4793-bbaa-c8dae197566d" containerName="cleanup" Oct 30 22:00:20.651092 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.650955 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa5fbac8-0bd8-4793-bbaa-c8dae197566d" containerName="cleanup" Oct 30 22:00:20.651092 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.650961 4560 cpu_manager.go:414] "containerMap: removing container" podUID="ae04a1f9-8997-42e2-b7e9-94e84fd97d71" containerName="cleanup" Oct 30 22:00:20.651092 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.650966 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae04a1f9-8997-42e2-b7e9-94e84fd97d71" containerName="cleanup" Oct 30 22:00:20.651092 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.650971 4560 cpu_manager.go:414] "containerMap: removing container" podUID="e174ecd3-9e50-458b-85aa-b130ceba6d70" containerName="cleanup" Oct 30 22:00:20.651092 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.650975 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e174ecd3-9e50-458b-85aa-b130ceba6d70" containerName="cleanup" Oct 30 22:00:20.651092 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.650981 4560 cpu_manager.go:414] "containerMap: removing container" podUID="aa5fbac8-0bd8-4793-bbaa-c8dae197566d" containerName="cleanup" Oct 30 22:00:20.651092 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.650984 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa5fbac8-0bd8-4793-bbaa-c8dae197566d" containerName="cleanup" Oct 30 22:00:20.651092 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.650999 4560 cpu_manager.go:414] "containerMap: removing container" podUID="d29a3644-c4e8-40eb-a34b-f531c2a1565b" containerName="cleanup" Oct 30 22:00:20.651092 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.651005 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d29a3644-c4e8-40eb-a34b-f531c2a1565b" containerName="cleanup" Oct 30 22:00:20.651092 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.651013 4560 cpu_manager.go:414] "containerMap: removing container" podUID="d29a3644-c4e8-40eb-a34b-f531c2a1565b" containerName="cleanup" Oct 30 22:00:20.651092 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.651018 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d29a3644-c4e8-40eb-a34b-f531c2a1565b" containerName="cleanup" Oct 30 22:00:20.651092 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.651032 4560 cpu_manager.go:414] "containerMap: removing container" podUID="a24ad108-152e-4690-82ae-71e98f901c61" containerName="cleanup" Oct 30 22:00:20.651092 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.651038 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a24ad108-152e-4690-82ae-71e98f901c61" containerName="cleanup" Oct 30 22:00:20.651092 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.651049 4560 cpu_manager.go:414] "containerMap: removing container" podUID="45e6d264-3503-493e-95f2-6308e04e93ab" containerName="cleanup" Oct 30 22:00:20.651092 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.651055 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45e6d264-3503-493e-95f2-6308e04e93ab" containerName="cleanup" Oct 30 22:00:20.651092 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.651075 4560 cpu_manager.go:414] "containerMap: removing container" podUID="08796630-f181-4b69-bd3c-32f627dff3ac" containerName="cleanup" Oct 30 22:00:20.651092 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.651082 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08796630-f181-4b69-bd3c-32f627dff3ac" containerName="cleanup" Oct 30 22:00:20.652056 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.651162 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="45e6d264-3503-493e-95f2-6308e04e93ab" containerName="cleanup" Oct 30 22:00:20.652056 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.651169 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae04a1f9-8997-42e2-b7e9-94e84fd97d71" containerName="cleanup" Oct 30 22:00:20.652056 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.651174 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="e174ecd3-9e50-458b-85aa-b130ceba6d70" containerName="cleanup" Oct 30 22:00:20.652056 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.651180 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa5fbac8-0bd8-4793-bbaa-c8dae197566d" containerName="cleanup" Oct 30 22:00:20.652056 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.651186 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa5fbac8-0bd8-4793-bbaa-c8dae197566d" containerName="cleanup" Oct 30 22:00:20.652056 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.651193 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="08796630-f181-4b69-bd3c-32f627dff3ac" containerName="cleanup" Oct 30 22:00:20.652056 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.651198 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa5fbac8-0bd8-4793-bbaa-c8dae197566d" containerName="cleanup" Oct 30 22:00:20.652056 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.651204 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="45e6d264-3503-493e-95f2-6308e04e93ab" containerName="cleanup" Oct 30 22:00:20.652056 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.651209 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="08796630-f181-4b69-bd3c-32f627dff3ac" containerName="cleanup" Oct 30 22:00:20.652056 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.651214 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae04a1f9-8997-42e2-b7e9-94e84fd97d71" containerName="cleanup" Oct 30 22:00:20.652056 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.651219 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="a24ad108-152e-4690-82ae-71e98f901c61" containerName="cleanup" Oct 30 22:00:20.652056 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.651225 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="d29a3644-c4e8-40eb-a34b-f531c2a1565b" containerName="cleanup" Oct 30 22:00:20.652056 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.651229 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="d29a3644-c4e8-40eb-a34b-f531c2a1565b" containerName="cleanup" Oct 30 22:00:20.652056 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.651235 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="a24ad108-152e-4690-82ae-71e98f901c61" containerName="cleanup" Oct 30 22:00:20.652056 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.651240 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="d29a3644-c4e8-40eb-a34b-f531c2a1565b" containerName="cleanup" Oct 30 22:00:20.652056 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.651245 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="08796630-f181-4b69-bd3c-32f627dff3ac" containerName="cleanup" Oct 30 22:00:20.652056 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.651251 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="a24ad108-152e-4690-82ae-71e98f901c61" containerName="cleanup" Oct 30 22:00:20.652056 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.651257 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="45e6d264-3503-493e-95f2-6308e04e93ab" containerName="cleanup" Oct 30 22:00:20.652056 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.651263 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="e174ecd3-9e50-458b-85aa-b130ceba6d70" containerName="cleanup" Oct 30 22:00:20.660316 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.660296 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" Oct 30 22:00:20.683193 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.683168 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889960-tfsx2" Oct 30 22:00:20.694326 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.694300 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g"] Oct 30 22:00:20.694741 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.694729 4560 cpu_manager.go:414] "containerMap: removing container" podUID="818309d7-73a4-404f-9942-5bcbb2a4fd14" containerName="cleanup" Oct 30 22:00:20.694779 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.694743 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="818309d7-73a4-404f-9942-5bcbb2a4fd14" containerName="cleanup" Oct 30 22:00:20.694779 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.694754 4560 cpu_manager.go:414] "containerMap: removing container" podUID="a24ad108-152e-4690-82ae-71e98f901c61" containerName="cleanup" Oct 30 22:00:20.694779 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.694760 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a24ad108-152e-4690-82ae-71e98f901c61" containerName="cleanup" Oct 30 22:00:20.694779 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.694772 4560 cpu_manager.go:414] "containerMap: removing container" podUID="08796630-f181-4b69-bd3c-32f627dff3ac" containerName="cleanup" Oct 30 22:00:20.694779 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.694778 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08796630-f181-4b69-bd3c-32f627dff3ac" containerName="cleanup" Oct 30 22:00:20.694976 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.694792 4560 cpu_manager.go:414] "containerMap: removing container" podUID="ae04a1f9-8997-42e2-b7e9-94e84fd97d71" containerName="cleanup" Oct 30 22:00:20.694976 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.694797 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae04a1f9-8997-42e2-b7e9-94e84fd97d71" containerName="cleanup" Oct 30 22:00:20.694976 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.694806 4560 cpu_manager.go:414] "containerMap: removing container" podUID="e174ecd3-9e50-458b-85aa-b130ceba6d70" containerName="cleanup" Oct 30 22:00:20.694976 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.694811 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e174ecd3-9e50-458b-85aa-b130ceba6d70" containerName="cleanup" Oct 30 22:00:20.694976 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.694818 4560 cpu_manager.go:414] "containerMap: removing container" podUID="aa5fbac8-0bd8-4793-bbaa-c8dae197566d" containerName="cleanup" Oct 30 22:00:20.694976 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.694822 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa5fbac8-0bd8-4793-bbaa-c8dae197566d" containerName="cleanup" Oct 30 22:00:20.694976 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.694836 4560 cpu_manager.go:414] "containerMap: removing container" podUID="818309d7-73a4-404f-9942-5bcbb2a4fd14" containerName="cleanup" Oct 30 22:00:20.694976 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.694840 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="818309d7-73a4-404f-9942-5bcbb2a4fd14" containerName="cleanup" Oct 30 22:00:20.694976 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.694846 4560 cpu_manager.go:414] "containerMap: removing container" podUID="818309d7-73a4-404f-9942-5bcbb2a4fd14" containerName="cleanup" Oct 30 22:00:20.694976 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.694850 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="818309d7-73a4-404f-9942-5bcbb2a4fd14" containerName="cleanup" Oct 30 22:00:20.694976 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.694855 4560 cpu_manager.go:414] "containerMap: removing container" podUID="45e6d264-3503-493e-95f2-6308e04e93ab" containerName="cleanup" Oct 30 22:00:20.694976 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.694859 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45e6d264-3503-493e-95f2-6308e04e93ab" containerName="cleanup" Oct 30 22:00:20.694976 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.694939 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="e174ecd3-9e50-458b-85aa-b130ceba6d70" containerName="cleanup" Oct 30 22:00:20.694976 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.694951 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="818309d7-73a4-404f-9942-5bcbb2a4fd14" containerName="cleanup" Oct 30 22:00:20.694976 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.694962 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae04a1f9-8997-42e2-b7e9-94e84fd97d71" containerName="cleanup" Oct 30 22:00:20.694976 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.694973 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="818309d7-73a4-404f-9942-5bcbb2a4fd14" containerName="cleanup" Oct 30 22:00:20.694976 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.694983 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="818309d7-73a4-404f-9942-5bcbb2a4fd14" containerName="cleanup" Oct 30 22:00:20.697763 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.697746 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45e6d264-3503-493e-95f2-6308e04e93ab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45e6d264-3503-493e-95f2-6308e04e93ab-maas-api-service-ca\") pod \"45e6d264-3503-493e-95f2-6308e04e93ab\" (UID: \"45e6d264-3503-493e-95f2-6308e04e93ab\") " Oct 30 22:00:20.697826 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.697797 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45e6d264-3503-493e-95f2-6308e04e93ab-kube-api-access-hdd9p\" (UniqueName: \"kubernetes.io/projected/45e6d264-3503-493e-95f2-6308e04e93ab-kube-api-access-hdd9p\") pod \"45e6d264-3503-493e-95f2-6308e04e93ab\" (UID: \"45e6d264-3503-493e-95f2-6308e04e93ab\") " Oct 30 22:00:20.698170 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.698136 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45e6d264-3503-493e-95f2-6308e04e93ab-maas-api-service-ca" pod "45e6d264-3503-493e-95f2-6308e04e93ab" (UID: "45e6d264-3503-493e-95f2-6308e04e93ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:20.698170 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.698158 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" Oct 30 22:00:20.700104 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.700081 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45e6d264-3503-493e-95f2-6308e04e93ab-kube-api-access-hdd9p" pod "45e6d264-3503-493e-95f2-6308e04e93ab" (UID: "45e6d264-3503-493e-95f2-6308e04e93ab"). InnerVolumeSpecName "kube-api-access-hdd9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:20.716307 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.716283 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p"] Oct 30 22:00:20.798822 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.798741 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g"] Oct 30 22:00:20.799106 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.799084 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/818309d7-73a4-404f-9942-5bcbb2a4fd14-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/818309d7-73a4-404f-9942-5bcbb2a4fd14-maas-api-service-ca\") pod \"818309d7-73a4-404f-9942-5bcbb2a4fd14\" (UID: \"818309d7-73a4-404f-9942-5bcbb2a4fd14\") " Oct 30 22:00:20.799275 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.799261 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/818309d7-73a4-404f-9942-5bcbb2a4fd14-kube-api-access-ptrhq\" (UniqueName: \"kubernetes.io/projected/818309d7-73a4-404f-9942-5bcbb2a4fd14-kube-api-access-ptrhq\") pod \"818309d7-73a4-404f-9942-5bcbb2a4fd14\" (UID: \"818309d7-73a4-404f-9942-5bcbb2a4fd14\") " Oct 30 22:00:20.799553 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.799516 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/818309d7-73a4-404f-9942-5bcbb2a4fd14-maas-api-service-ca" pod "818309d7-73a4-404f-9942-5bcbb2a4fd14" (UID: "818309d7-73a4-404f-9942-5bcbb2a4fd14"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:20.799673 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.799651 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-workload-socket\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g\" (UID: \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" Oct 30 22:00:20.799751 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.799681 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-istio-envoy\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g\" (UID: \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" Oct 30 22:00:20.799751 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.799706 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnjx2\" (UniqueName: \"kubernetes.io/projected/f87e7e5a-9553-4dcb-9efa-d6557a171331-kube-api-access-xnjx2\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p\" (UID: \"f87e7e5a-9553-4dcb-9efa-d6557a171331\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" Oct 30 22:00:20.799857 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.799765 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-istio-podinfo\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g\" (UID: \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" Oct 30 22:00:20.799897 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.799869 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-istiod-ca-cert\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g\" (UID: \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" Oct 30 22:00:20.799939 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.799922 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f87e7e5a-9553-4dcb-9efa-d6557a171331-istio-envoy\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p\" (UID: \"f87e7e5a-9553-4dcb-9efa-d6557a171331\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" Oct 30 22:00:20.799989 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.799950 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-workload-certs\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g\" (UID: \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" Oct 30 22:00:20.800031 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.799987 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bc8f\" (UniqueName: \"kubernetes.io/projected/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-kube-api-access-4bc8f\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g\" (UID: \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" Oct 30 22:00:20.800031 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.800010 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f87e7e5a-9553-4dcb-9efa-d6557a171331-istiod-ca-cert\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p\" (UID: \"f87e7e5a-9553-4dcb-9efa-d6557a171331\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" Oct 30 22:00:20.800136 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.800036 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-istio-token\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g\" (UID: \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" Oct 30 22:00:20.800179 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.800133 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f87e7e5a-9553-4dcb-9efa-d6557a171331-workload-certs\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p\" (UID: \"f87e7e5a-9553-4dcb-9efa-d6557a171331\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" Oct 30 22:00:20.800267 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.800244 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f87e7e5a-9553-4dcb-9efa-d6557a171331-istio-token\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p\" (UID: \"f87e7e5a-9553-4dcb-9efa-d6557a171331\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" Oct 30 22:00:20.800300 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.800282 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-credential-socket\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g\" (UID: \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" Oct 30 22:00:20.800332 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.800306 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f87e7e5a-9553-4dcb-9efa-d6557a171331-istio-data\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p\" (UID: \"f87e7e5a-9553-4dcb-9efa-d6557a171331\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" Oct 30 22:00:20.800393 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.800366 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f87e7e5a-9553-4dcb-9efa-d6557a171331-workload-socket\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p\" (UID: \"f87e7e5a-9553-4dcb-9efa-d6557a171331\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" Oct 30 22:00:20.800447 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.800401 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-istio-data\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g\" (UID: \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" Oct 30 22:00:20.800904 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.800877 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f87e7e5a-9553-4dcb-9efa-d6557a171331-istio-podinfo\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p\" (UID: \"f87e7e5a-9553-4dcb-9efa-d6557a171331\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" Oct 30 22:00:20.801018 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.800920 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f87e7e5a-9553-4dcb-9efa-d6557a171331-credential-socket\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p\" (UID: \"f87e7e5a-9553-4dcb-9efa-d6557a171331\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" Oct 30 22:00:20.801018 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.800998 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45e6d264-3503-493e-95f2-6308e04e93ab-maas-api-service-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:20.801132 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.801031 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdd9p\" (UniqueName: \"kubernetes.io/projected/45e6d264-3503-493e-95f2-6308e04e93ab-kube-api-access-hdd9p\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:20.801132 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.801060 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/818309d7-73a4-404f-9942-5bcbb2a4fd14-maas-api-service-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:20.802051 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.802030 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/818309d7-73a4-404f-9942-5bcbb2a4fd14-kube-api-access-ptrhq" pod "818309d7-73a4-404f-9942-5bcbb2a4fd14" (UID: "818309d7-73a4-404f-9942-5bcbb2a4fd14"). InnerVolumeSpecName "kube-api-access-ptrhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:20.877436 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.877398 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889960-tfsx2" event={"ID":"818309d7-73a4-404f-9942-5bcbb2a4fd14","Type":"ContainerDied","Data":"f1b4a50a0245f1214ee07b38a5fd309501b73f644432a08bf8b639f8ac97ec79"} Oct 30 22:00:20.877880 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.877412 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889960-tfsx2" Oct 30 22:00:20.877880 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.877481 4560 scope.go:122] "RemoveContainer" containerID="3cd924f24af4d7b736dd657432335eb960a31f73ee86b9d2356758e6b76333c4" Oct 30 22:00:20.883256 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.883238 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889960-g9zpz" Oct 30 22:00:20.883398 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.883231 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889960-g9zpz" event={"ID":"45e6d264-3503-493e-95f2-6308e04e93ab","Type":"ContainerDied","Data":"f5a024db04d55021e6d3dd201b5932cefc01462dc3c84f7fb8b121dd15f68b99"} Oct 30 22:00:20.888034 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.888009 4560 scope.go:122] "RemoveContainer" containerID="eef10b579a71d82c82297491c17065d1a714b5a122f7e786e33d011cf30f0815" Oct 30 22:00:20.902689 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.902670 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-workload-socket\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g\" (UID: \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" Oct 30 22:00:20.902773 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.902698 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-istio-envoy\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g\" (UID: \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" Oct 30 22:00:20.902773 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.902718 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnjx2\" (UniqueName: \"kubernetes.io/projected/f87e7e5a-9553-4dcb-9efa-d6557a171331-kube-api-access-xnjx2\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p\" (UID: \"f87e7e5a-9553-4dcb-9efa-d6557a171331\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" Oct 30 22:00:20.902773 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.902733 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-istio-podinfo\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g\" (UID: \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" Oct 30 22:00:20.902893 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.902778 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-istiod-ca-cert\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g\" (UID: \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" Oct 30 22:00:20.902893 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.902800 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f87e7e5a-9553-4dcb-9efa-d6557a171331-istio-envoy\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p\" (UID: \"f87e7e5a-9553-4dcb-9efa-d6557a171331\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" Oct 30 22:00:20.902893 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.902823 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-workload-certs\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g\" (UID: \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" Oct 30 22:00:20.902893 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.902862 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bc8f\" (UniqueName: \"kubernetes.io/projected/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-kube-api-access-4bc8f\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g\" (UID: \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" Oct 30 22:00:20.902893 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.902887 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f87e7e5a-9553-4dcb-9efa-d6557a171331-istiod-ca-cert\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p\" (UID: \"f87e7e5a-9553-4dcb-9efa-d6557a171331\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" Oct 30 22:00:20.903145 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.902916 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-istio-token\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g\" (UID: \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" Oct 30 22:00:20.903145 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.902961 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f87e7e5a-9553-4dcb-9efa-d6557a171331-workload-certs\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p\" (UID: \"f87e7e5a-9553-4dcb-9efa-d6557a171331\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" Oct 30 22:00:20.903145 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.902997 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f87e7e5a-9553-4dcb-9efa-d6557a171331-istio-token\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p\" (UID: \"f87e7e5a-9553-4dcb-9efa-d6557a171331\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" Oct 30 22:00:20.903145 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.903022 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-credential-socket\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g\" (UID: \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" Oct 30 22:00:20.903145 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.903050 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f87e7e5a-9553-4dcb-9efa-d6557a171331-istio-data\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p\" (UID: \"f87e7e5a-9553-4dcb-9efa-d6557a171331\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" Oct 30 22:00:20.903145 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.903079 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-workload-socket\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g\" (UID: \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" Oct 30 22:00:20.903145 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.903085 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f87e7e5a-9553-4dcb-9efa-d6557a171331-workload-socket\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p\" (UID: \"f87e7e5a-9553-4dcb-9efa-d6557a171331\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" Oct 30 22:00:20.903535 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.903158 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-istio-data\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g\" (UID: \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" Oct 30 22:00:20.903535 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.903209 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f87e7e5a-9553-4dcb-9efa-d6557a171331-istio-podinfo\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p\" (UID: \"f87e7e5a-9553-4dcb-9efa-d6557a171331\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" Oct 30 22:00:20.903535 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.903247 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f87e7e5a-9553-4dcb-9efa-d6557a171331-credential-socket\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p\" (UID: \"f87e7e5a-9553-4dcb-9efa-d6557a171331\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" Oct 30 22:00:20.903535 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.903360 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f87e7e5a-9553-4dcb-9efa-d6557a171331-workload-socket\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p\" (UID: \"f87e7e5a-9553-4dcb-9efa-d6557a171331\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" Oct 30 22:00:20.903535 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.903447 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-workload-certs\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g\" (UID: \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" Oct 30 22:00:20.903535 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.903501 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptrhq\" (UniqueName: \"kubernetes.io/projected/818309d7-73a4-404f-9942-5bcbb2a4fd14-kube-api-access-ptrhq\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:20.903907 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.903682 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-istio-data\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g\" (UID: \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" Oct 30 22:00:20.903907 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.903726 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f87e7e5a-9553-4dcb-9efa-d6557a171331-credential-socket\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p\" (UID: \"f87e7e5a-9553-4dcb-9efa-d6557a171331\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" Oct 30 22:00:20.903907 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.903754 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-istiod-ca-cert\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g\" (UID: \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" Oct 30 22:00:20.904085 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.903949 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f87e7e5a-9553-4dcb-9efa-d6557a171331-workload-certs\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p\" (UID: \"f87e7e5a-9553-4dcb-9efa-d6557a171331\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" Oct 30 22:00:20.904085 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.904009 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f87e7e5a-9553-4dcb-9efa-d6557a171331-istiod-ca-cert\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p\" (UID: \"f87e7e5a-9553-4dcb-9efa-d6557a171331\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" Oct 30 22:00:20.904085 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.904043 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-credential-socket\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g\" (UID: \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" Oct 30 22:00:20.904229 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.904161 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f87e7e5a-9553-4dcb-9efa-d6557a171331-istio-data\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p\" (UID: \"f87e7e5a-9553-4dcb-9efa-d6557a171331\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" Oct 30 22:00:20.905177 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.905154 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-istio-envoy\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g\" (UID: \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" Oct 30 22:00:20.905488 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.905465 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f87e7e5a-9553-4dcb-9efa-d6557a171331-istio-envoy\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p\" (UID: \"f87e7e5a-9553-4dcb-9efa-d6557a171331\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" Oct 30 22:00:20.905873 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.905857 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f87e7e5a-9553-4dcb-9efa-d6557a171331-istio-podinfo\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p\" (UID: \"f87e7e5a-9553-4dcb-9efa-d6557a171331\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" Oct 30 22:00:20.906298 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.906278 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-istio-podinfo\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g\" (UID: \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" Oct 30 22:00:20.969932 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.961695 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnjx2\" (UniqueName: \"kubernetes.io/projected/f87e7e5a-9553-4dcb-9efa-d6557a171331-kube-api-access-xnjx2\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p\" (UID: \"f87e7e5a-9553-4dcb-9efa-d6557a171331\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" Oct 30 22:00:20.969932 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.964590 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f87e7e5a-9553-4dcb-9efa-d6557a171331-istio-token\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p\" (UID: \"f87e7e5a-9553-4dcb-9efa-d6557a171331\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" Oct 30 22:00:20.969932 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.964818 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-istio-token\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g\" (UID: \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" Oct 30 22:00:20.970802 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.970769 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bc8f\" (UniqueName: \"kubernetes.io/projected/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-kube-api-access-4bc8f\") pod \"e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g\" (UID: \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\") " pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" Oct 30 22:00:20.991481 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:20.991454 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" Oct 30 22:00:21.020900 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.020874 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" Oct 30 22:00:21.091244 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.091163 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889960-tfsx2"] Oct 30 22:00:21.094526 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.094500 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-554f46-29889960-tfsx2"] Oct 30 22:00:21.149083 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.149053 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ca0a55-29889960-8wsx6"] Oct 30 22:00:21.271994 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.271972 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ca0a55-29889960-8wsx6" Oct 30 22:00:21.310393 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.310313 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889960-g9zpz"] Oct 30 22:00:21.315124 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.315100 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71aab5-29889960-g9zpz"] Oct 30 22:00:21.389106 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.389079 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bfdf30b9-29889960-4c9wb"] Oct 30 22:00:21.409504 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.409474 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8a90e7a-b379-4742-a167-01aabd410bf9-kube-api-access-g2z4c\" (UniqueName: \"kubernetes.io/projected/d8a90e7a-b379-4742-a167-01aabd410bf9-kube-api-access-g2z4c\") pod \"d8a90e7a-b379-4742-a167-01aabd410bf9\" (UID: \"d8a90e7a-b379-4742-a167-01aabd410bf9\") " Oct 30 22:00:21.409652 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.409615 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8a90e7a-b379-4742-a167-01aabd410bf9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a90e7a-b379-4742-a167-01aabd410bf9-maas-api-service-ca\") pod \"d8a90e7a-b379-4742-a167-01aabd410bf9\" (UID: \"d8a90e7a-b379-4742-a167-01aabd410bf9\") " Oct 30 22:00:21.409982 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.409960 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8a90e7a-b379-4742-a167-01aabd410bf9-maas-api-service-ca" pod "d8a90e7a-b379-4742-a167-01aabd410bf9" (UID: "d8a90e7a-b379-4742-a167-01aabd410bf9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:21.411704 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.411682 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8a90e7a-b379-4742-a167-01aabd410bf9-kube-api-access-g2z4c" pod "d8a90e7a-b379-4742-a167-01aabd410bf9" (UID: "d8a90e7a-b379-4742-a167-01aabd410bf9"). InnerVolumeSpecName "kube-api-access-g2z4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:21.464491 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.464194 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:00:21.464491 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.464364 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:00:21.464491 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.464462 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:00:21.465285 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.465200 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g"] Oct 30 22:00:21.468746 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.468700 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:00:21.469105 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.468855 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:00:21.469105 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.468993 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:00:21.472075 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.472053 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p"] Oct 30 22:00:21.511365 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.511074 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2z4c\" (UniqueName: \"kubernetes.io/projected/d8a90e7a-b379-4742-a167-01aabd410bf9-kube-api-access-g2z4c\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:21.511365 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.511106 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a90e7a-b379-4742-a167-01aabd410bf9-maas-api-service-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:21.549837 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.549813 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bfdf30b9-29889960-4c9wb" Oct 30 22:00:21.611845 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.611730 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22606928-4bc3-49e5-91e9-00497b022b10-kube-api-access-wppj5\" (UniqueName: \"kubernetes.io/projected/22606928-4bc3-49e5-91e9-00497b022b10-kube-api-access-wppj5\") pod \"22606928-4bc3-49e5-91e9-00497b022b10\" (UID: \"22606928-4bc3-49e5-91e9-00497b022b10\") " Oct 30 22:00:21.611845 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.611803 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22606928-4bc3-49e5-91e9-00497b022b10-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22606928-4bc3-49e5-91e9-00497b022b10-maas-api-service-ca\") pod \"22606928-4bc3-49e5-91e9-00497b022b10\" (UID: \"22606928-4bc3-49e5-91e9-00497b022b10\") " Oct 30 22:00:21.612252 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.612221 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22606928-4bc3-49e5-91e9-00497b022b10-maas-api-service-ca" pod "22606928-4bc3-49e5-91e9-00497b022b10" (UID: "22606928-4bc3-49e5-91e9-00497b022b10"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:21.612553 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.612533 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22606928-4bc3-49e5-91e9-00497b022b10-maas-api-service-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:21.613978 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.613942 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22606928-4bc3-49e5-91e9-00497b022b10-kube-api-access-wppj5" pod "22606928-4bc3-49e5-91e9-00497b022b10" (UID: "22606928-4bc3-49e5-91e9-00497b022b10"). InnerVolumeSpecName "kube-api-access-wppj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:21.714406 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.714373 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wppj5\" (UniqueName: \"kubernetes.io/projected/22606928-4bc3-49e5-91e9-00497b022b10-kube-api-access-wppj5\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:21.891403 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.891287 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" event={"ID":"1e67aa41-ff98-4eab-a76c-020b1de1ff8e","Type":"ContainerStarted","Data":"587c91fe2d9dbdea9b0b8ac06160d64e0ee973448aa444a51e8880ab1a59d8e0"} Oct 30 22:00:21.891403 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.891377 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" event={"ID":"1e67aa41-ff98-4eab-a76c-020b1de1ff8e","Type":"ContainerStarted","Data":"ee07d53d88fa7b37cd2fc3203cb1751dbb11cc60ca5579cb23a255ddcc642fba"} Oct 30 22:00:21.893571 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.893541 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" event={"ID":"f87e7e5a-9553-4dcb-9efa-d6557a171331","Type":"ContainerStarted","Data":"5f1b662cc15575130db940c549390e26b013715fd041c3df354f1a335a1853ab"} Oct 30 22:00:21.893713 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.893598 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" event={"ID":"f87e7e5a-9553-4dcb-9efa-d6557a171331","Type":"ContainerStarted","Data":"f38d71cac9388679d2afb166df1dee04b9d9034f5d4479a4999e572018420ece"} Oct 30 22:00:21.896079 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.896055 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ca0a55-29889960-8wsx6" Oct 30 22:00:21.896208 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.896051 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ca0a55-29889960-8wsx6" event={"ID":"d8a90e7a-b379-4742-a167-01aabd410bf9","Type":"ContainerDied","Data":"9f0a876bfd508670a4e144a241a2003a7cd77ef8283c6934c3263cb98edc57b7"} Oct 30 22:00:21.896272 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.896209 4560 scope.go:122] "RemoveContainer" containerID="b440b6c4d46b6acecc67671f6fba1a1433ccccfc119b90e6416ebca602a8dd69" Oct 30 22:00:21.897758 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.897729 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bfdf30b9-29889960-4c9wb" event={"ID":"22606928-4bc3-49e5-91e9-00497b022b10","Type":"ContainerDied","Data":"8bcb156bbbd327b814310f20b73893ebde42ece43c31cc0f7a7886ab36a857a6"} Oct 30 22:00:21.897854 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.897741 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bfdf30b9-29889960-4c9wb" Oct 30 22:00:21.906751 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.906729 4560 scope.go:122] "RemoveContainer" containerID="117d50ae00eccab7b8fe9e45b10f09767775de6a8ad9841886c2c006ece320d8" Oct 30 22:00:21.992237 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:21.992208 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" Oct 30 22:00:22.021785 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:22.021757 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" Oct 30 22:00:22.023130 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:22.023106 4560 patch_prober.go:29] interesting pod/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Oct 30 22:00:22.023210 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:22.023149 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" podUID="1e67aa41-ff98-4eab-a76c-020b1de1ff8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Oct 30 22:00:22.025795 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:22.025750 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" podStartSLOduration=2.02573456 podStartE2EDuration="2.02573456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:22.023877595 +0000 UTC m=+2797.578577372" watchObservedRunningTime="2026-10-30 22:00:22.02573456 +0000 UTC m=+2797.580434391" Oct 30 22:00:22.176548 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:22.176442 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" podStartSLOduration=2.176424171 podStartE2EDuration="2.176424171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:22.175052102 +0000 UTC m=+2797.729751879" watchObservedRunningTime="2026-10-30 22:00:22.176424171 +0000 UTC m=+2797.731123949" Oct 30 22:00:22.293361 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:22.292111 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ca0a55-29889960-8wsx6"] Oct 30 22:00:22.296785 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:22.296761 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ca0a55-29889960-8wsx6"] Oct 30 22:00:22.418967 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:22.418928 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bfdf30b9-29889960-4c9wb"] Oct 30 22:00:22.445773 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:22.445675 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bfdf30b9-29889960-4c9wb"] Oct 30 22:00:22.583442 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:22.583411 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22606928-4bc3-49e5-91e9-00497b022b10" path="/var/lib/kubelet/pods/22606928-4bc3-49e5-91e9-00497b022b10/volumes" Oct 30 22:00:22.583926 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:22.583910 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45e6d264-3503-493e-95f2-6308e04e93ab" path="/var/lib/kubelet/pods/45e6d264-3503-493e-95f2-6308e04e93ab/volumes" Oct 30 22:00:22.584391 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:22.584374 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="818309d7-73a4-404f-9942-5bcbb2a4fd14" path="/var/lib/kubelet/pods/818309d7-73a4-404f-9942-5bcbb2a4fd14/volumes" Oct 30 22:00:22.585106 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:22.585088 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8a90e7a-b379-4742-a167-01aabd410bf9" path="/var/lib/kubelet/pods/d8a90e7a-b379-4742-a167-01aabd410bf9/volumes" Oct 30 22:00:22.993547 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:22.993518 4560 patch_prober.go:29] interesting pod/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 30 22:00:22.994040 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:22.993568 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" podUID="f87e7e5a-9553-4dcb-9efa-d6557a171331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 30 22:00:23.021175 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:23.021143 4560 patch_prober.go:29] interesting pod/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Oct 30 22:00:23.021292 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:23.021196 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" podUID="1e67aa41-ff98-4eab-a76c-020b1de1ff8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Oct 30 22:00:23.991910 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:23.991876 4560 patch_prober.go:29] interesting pod/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 30 22:00:23.992089 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:23.991930 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" podUID="f87e7e5a-9553-4dcb-9efa-d6557a171331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 30 22:00:24.022076 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:24.022046 4560 patch_prober.go:29] interesting pod/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Oct 30 22:00:24.022447 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:24.022098 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" podUID="1e67aa41-ff98-4eab-a76c-020b1de1ff8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Oct 30 22:00:24.140376 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:24.140134 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:00:24.991932 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:24.991883 4560 patch_prober.go:29] interesting pod/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 30 22:00:24.992145 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:24.991961 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" podUID="f87e7e5a-9553-4dcb-9efa-d6557a171331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 30 22:00:25.021233 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:25.021200 4560 patch_prober.go:29] interesting pod/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Oct 30 22:00:25.021382 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:25.021245 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" podUID="1e67aa41-ff98-4eab-a76c-020b1de1ff8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Oct 30 22:00:25.992168 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:25.992135 4560 patch_prober.go:29] interesting pod/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 30 22:00:25.992656 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:25.992188 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" podUID="f87e7e5a-9553-4dcb-9efa-d6557a171331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 30 22:00:26.021280 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:26.021242 4560 patch_prober.go:29] interesting pod/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Oct 30 22:00:26.021464 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:26.021311 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" podUID="1e67aa41-ff98-4eab-a76c-020b1de1ff8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Oct 30 22:00:26.992629 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:26.992592 4560 patch_prober.go:29] interesting pod/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 30 22:00:26.993109 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:26.992670 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" podUID="f87e7e5a-9553-4dcb-9efa-d6557a171331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 30 22:00:27.021422 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:27.021389 4560 patch_prober.go:29] interesting pod/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Oct 30 22:00:27.021568 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:27.021440 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" podUID="1e67aa41-ff98-4eab-a76c-020b1de1ff8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Oct 30 22:00:27.992457 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:27.992418 4560 patch_prober.go:29] interesting pod/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 30 22:00:27.992725 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:27.992696 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" podUID="f87e7e5a-9553-4dcb-9efa-d6557a171331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 30 22:00:28.022270 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:28.022225 4560 patch_prober.go:29] interesting pod/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Oct 30 22:00:28.022443 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:28.022297 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" podUID="1e67aa41-ff98-4eab-a76c-020b1de1ff8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Oct 30 22:00:28.996529 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:28.996499 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" Oct 30 22:00:28.996955 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:28.996851 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" Oct 30 22:00:28.997653 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:28.997636 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" Oct 30 22:00:29.026294 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:29.026274 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" Oct 30 22:00:29.026575 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:29.026554 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" Oct 30 22:00:29.027142 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:29.027120 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" Oct 30 22:00:34.762272 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:34.762224 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-554f46-trdg6"] Oct 30 22:00:34.811027 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:34.810981 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-554f46-trdg6"] Oct 30 22:00:35.959810 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:35.959777 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ed6135-4hx4s"] Oct 30 22:00:36.005868 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:36.005838 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ed6135-4hx4s"] Oct 30 22:00:36.576882 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:36.576849 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4556ce3a-898e-49ee-8838-3446807183fa" path="/var/lib/kubelet/pods/4556ce3a-898e-49ee-8838-3446807183fa/volumes" Oct 30 22:00:36.577327 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:36.577314 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a42c755a-8283-4f86-86f4-9bdbe1753bdd" path="/var/lib/kubelet/pods/a42c755a-8283-4f86-86f4-9bdbe1753bdd/volumes" Oct 30 22:00:43.342648 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:43.342602 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b"] Oct 30 22:00:43.343110 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:43.342821 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" podUID="8e1ac4c8-c9d4-4419-8a37-cabedad5ad01" containerName="istio-proxy" containerID="cri-o://2713788d053dd37274192bceae58790d357dc83ecdb7f696140b4ebded3d85da" gracePeriod=30 Oct 30 22:00:43.444147 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:43.444098 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-kfkwt"] Oct 30 22:00:43.457254 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:43.457216 4560 cpu_manager.go:414] "containerMap: removing container" podUID="22606928-4bc3-49e5-91e9-00497b022b10" containerName="cleanup" Oct 30 22:00:43.457513 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:43.457491 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22606928-4bc3-49e5-91e9-00497b022b10" containerName="cleanup" Oct 30 22:00:43.457690 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:43.457677 4560 cpu_manager.go:414] "containerMap: removing container" podUID="22606928-4bc3-49e5-91e9-00497b022b10" containerName="cleanup" Oct 30 22:00:43.457778 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:43.457767 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22606928-4bc3-49e5-91e9-00497b022b10" containerName="cleanup" Oct 30 22:00:43.457858 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:43.457849 4560 cpu_manager.go:414] "containerMap: removing container" podUID="d8a90e7a-b379-4742-a167-01aabd410bf9" containerName="cleanup" Oct 30 22:00:43.457934 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:43.457924 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8a90e7a-b379-4742-a167-01aabd410bf9" containerName="cleanup" Oct 30 22:00:43.458050 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:43.458040 4560 cpu_manager.go:414] "containerMap: removing container" podUID="d8a90e7a-b379-4742-a167-01aabd410bf9" containerName="cleanup" Oct 30 22:00:43.458126 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:43.458116 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8a90e7a-b379-4742-a167-01aabd410bf9" containerName="cleanup" Oct 30 22:00:43.458401 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:43.458384 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="22606928-4bc3-49e5-91e9-00497b022b10" containerName="cleanup" Oct 30 22:00:43.458522 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:43.458511 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8a90e7a-b379-4742-a167-01aabd410bf9" containerName="cleanup" Oct 30 22:00:43.464414 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:43.464389 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8a90e7a-b379-4742-a167-01aabd410bf9" containerName="cleanup" Oct 30 22:00:43.464618 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:43.464604 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="22606928-4bc3-49e5-91e9-00497b022b10" containerName="cleanup" Oct 30 22:00:43.464798 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:43.464785 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="22606928-4bc3-49e5-91e9-00497b022b10" containerName="cleanup" Oct 30 22:00:43.464895 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:43.464883 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8a90e7a-b379-4742-a167-01aabd410bf9" containerName="cleanup" Oct 30 22:00:43.482364 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:43.474611 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-kfkwt" Oct 30 22:00:43.512361 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:43.507388 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-kfkwt"] Oct 30 22:00:43.548842 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:43.524382 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d5d0829-37d1-4139-8c40-6c7cc3ba92b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-bfdf30b9-kfkwt\" (UID: \"6d5d0829-37d1-4139-8c40-6c7cc3ba92b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-kfkwt" Oct 30 22:00:43.548842 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:43.548561 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrqwc\" (UniqueName: \"kubernetes.io/projected/6d5d0829-37d1-4139-8c40-6c7cc3ba92b1-kube-api-access-nrqwc\") pod \"maas-api-revoke-keys-e2e-mt-bfdf30b9-kfkwt\" (UID: \"6d5d0829-37d1-4139-8c40-6c7cc3ba92b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-kfkwt" Oct 30 22:00:43.548842 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:43.548649 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d5d0829-37d1-4139-8c40-6c7cc3ba92b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-bfdf30b9-kfkwt\" (UID: \"6d5d0829-37d1-4139-8c40-6c7cc3ba92b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-kfkwt" Oct 30 22:00:43.556592 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:43.556557 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:00:43.556774 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:43.556636 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:00:43.556774 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:43.556728 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:00:43.556901 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:43.556825 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:00:43.649814 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:43.649738 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d5d0829-37d1-4139-8c40-6c7cc3ba92b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-bfdf30b9-kfkwt\" (UID: \"6d5d0829-37d1-4139-8c40-6c7cc3ba92b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-kfkwt" Oct 30 22:00:43.649972 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:43.649820 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrqwc\" (UniqueName: \"kubernetes.io/projected/6d5d0829-37d1-4139-8c40-6c7cc3ba92b1-kube-api-access-nrqwc\") pod \"maas-api-revoke-keys-e2e-mt-bfdf30b9-kfkwt\" (UID: \"6d5d0829-37d1-4139-8c40-6c7cc3ba92b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-kfkwt" Oct 30 22:00:43.649972 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:43.649854 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d5d0829-37d1-4139-8c40-6c7cc3ba92b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-bfdf30b9-kfkwt\" (UID: \"6d5d0829-37d1-4139-8c40-6c7cc3ba92b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-kfkwt" Oct 30 22:00:43.650457 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:43.650433 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d5d0829-37d1-4139-8c40-6c7cc3ba92b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-bfdf30b9-kfkwt\" (UID: \"6d5d0829-37d1-4139-8c40-6c7cc3ba92b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-kfkwt" Oct 30 22:00:43.652189 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:43.652169 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d5d0829-37d1-4139-8c40-6c7cc3ba92b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-bfdf30b9-kfkwt\" (UID: \"6d5d0829-37d1-4139-8c40-6c7cc3ba92b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-kfkwt" Oct 30 22:00:43.707667 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:43.707644 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrqwc\" (UniqueName: \"kubernetes.io/projected/6d5d0829-37d1-4139-8c40-6c7cc3ba92b1-kube-api-access-nrqwc\") pod \"maas-api-revoke-keys-e2e-mt-bfdf30b9-kfkwt\" (UID: \"6d5d0829-37d1-4139-8c40-6c7cc3ba92b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-kfkwt" Oct 30 22:00:43.860246 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:43.860213 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-kfkwt" Oct 30 22:00:44.234630 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:44.234597 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-kfkwt"] Oct 30 22:00:44.239811 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:00:44.239783 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d5d0829_37d1_4139_8c40_6c7cc3ba92b1.slice/crio-a5c305ac25ea1ced5985b7a91f45482c139f0b6c3cd0bdd2686ed40839080a41 WatchSource:0}: Error finding container a5c305ac25ea1ced5985b7a91f45482c139f0b6c3cd0bdd2686ed40839080a41: Status 404 returned error can't find the container with id a5c305ac25ea1ced5985b7a91f45482c139f0b6c3cd0bdd2686ed40839080a41 Oct 30 22:00:45.140897 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:45.140851 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-kfkwt" event={"ID":"6d5d0829-37d1-4139-8c40-6c7cc3ba92b1","Type":"ContainerStarted","Data":"36c6711f316c4ebe76ae889fa17ee363a0954b6d4ce27043f5fb27ae40ea59b1"} Oct 30 22:00:45.146828 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:45.146791 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-kfkwt" event={"ID":"6d5d0829-37d1-4139-8c40-6c7cc3ba92b1","Type":"ContainerStarted","Data":"a5c305ac25ea1ced5985b7a91f45482c139f0b6c3cd0bdd2686ed40839080a41"} Oct 30 22:00:45.258512 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:45.258459 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-kfkwt" podStartSLOduration=2.258445302 podStartE2EDuration="2.258445302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:45.256862095 +0000 UTC m=+2820.811561873" watchObservedRunningTime="2026-10-30 22:00:45.258445302 +0000 UTC m=+2820.813145078" Oct 30 22:00:45.919996 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:45.919966 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71aab5-rt94x"] Oct 30 22:00:45.920553 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:45.920525 4560 cpu_manager.go:414] "containerMap: removing container" podUID="d8a90e7a-b379-4742-a167-01aabd410bf9" containerName="cleanup" Oct 30 22:00:45.920553 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:45.920544 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8a90e7a-b379-4742-a167-01aabd410bf9" containerName="cleanup" Oct 30 22:00:45.920720 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:45.920574 4560 cpu_manager.go:414] "containerMap: removing container" podUID="22606928-4bc3-49e5-91e9-00497b022b10" containerName="cleanup" Oct 30 22:00:45.920720 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:45.920579 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22606928-4bc3-49e5-91e9-00497b022b10" containerName="cleanup" Oct 30 22:00:45.923643 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:45.923627 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71aab5-rt94x" Oct 30 22:00:45.979019 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:45.978988 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71aab5-rt94x"] Oct 30 22:00:45.979692 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:45.979665 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wbdm\" (UniqueName: \"kubernetes.io/projected/363ad13f-c717-4b4e-a4fc-f114c21b5d8f-kube-api-access-7wbdm\") pod \"maas-api-revoke-keys-e2e-worker-w3-71aab5-rt94x\" (UID: \"363ad13f-c717-4b4e-a4fc-f114c21b5d8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71aab5-rt94x" Oct 30 22:00:45.979928 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:45.979900 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363ad13f-c717-4b4e-a4fc-f114c21b5d8f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-71aab5-rt94x\" (UID: \"363ad13f-c717-4b4e-a4fc-f114c21b5d8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71aab5-rt94x" Oct 30 22:00:45.980031 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:45.980003 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363ad13f-c717-4b4e-a4fc-f114c21b5d8f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-71aab5-rt94x\" (UID: \"363ad13f-c717-4b4e-a4fc-f114c21b5d8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71aab5-rt94x" Oct 30 22:00:46.081102 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:46.081071 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363ad13f-c717-4b4e-a4fc-f114c21b5d8f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-71aab5-rt94x\" (UID: \"363ad13f-c717-4b4e-a4fc-f114c21b5d8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71aab5-rt94x" Oct 30 22:00:46.081263 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:46.081134 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363ad13f-c717-4b4e-a4fc-f114c21b5d8f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-71aab5-rt94x\" (UID: \"363ad13f-c717-4b4e-a4fc-f114c21b5d8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71aab5-rt94x" Oct 30 22:00:46.081263 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:46.081169 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wbdm\" (UniqueName: \"kubernetes.io/projected/363ad13f-c717-4b4e-a4fc-f114c21b5d8f-kube-api-access-7wbdm\") pod \"maas-api-revoke-keys-e2e-worker-w3-71aab5-rt94x\" (UID: \"363ad13f-c717-4b4e-a4fc-f114c21b5d8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71aab5-rt94x" Oct 30 22:00:46.081719 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:46.081690 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363ad13f-c717-4b4e-a4fc-f114c21b5d8f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-71aab5-rt94x\" (UID: \"363ad13f-c717-4b4e-a4fc-f114c21b5d8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71aab5-rt94x" Oct 30 22:00:46.083422 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:46.083400 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363ad13f-c717-4b4e-a4fc-f114c21b5d8f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-71aab5-rt94x\" (UID: \"363ad13f-c717-4b4e-a4fc-f114c21b5d8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71aab5-rt94x" Oct 30 22:00:46.139212 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:46.139184 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wbdm\" (UniqueName: \"kubernetes.io/projected/363ad13f-c717-4b4e-a4fc-f114c21b5d8f-kube-api-access-7wbdm\") pod \"maas-api-revoke-keys-e2e-worker-w3-71aab5-rt94x\" (UID: \"363ad13f-c717-4b4e-a4fc-f114c21b5d8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71aab5-rt94x" Oct 30 22:00:46.151726 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:46.151697 4560 generic.go:356] "Generic (PLEG): container finished" podID="6d5d0829-37d1-4139-8c40-6c7cc3ba92b1" containerID="36c6711f316c4ebe76ae889fa17ee363a0954b6d4ce27043f5fb27ae40ea59b1" exitCode=0 Oct 30 22:00:46.152036 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:46.151777 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-kfkwt" event={"ID":"6d5d0829-37d1-4139-8c40-6c7cc3ba92b1","Type":"ContainerDied","Data":"36c6711f316c4ebe76ae889fa17ee363a0954b6d4ce27043f5fb27ae40ea59b1"} Oct 30 22:00:46.236407 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:46.236380 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71aab5-rt94x" Oct 30 22:00:46.619757 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:46.619683 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71aab5-rt94x"] Oct 30 22:00:47.160011 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:47.159970 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71aab5-rt94x" event={"ID":"363ad13f-c717-4b4e-a4fc-f114c21b5d8f","Type":"ContainerStarted","Data":"a445c0223c04cd4fb73dda7a38014e56f31062b45ea7ecb33faa691ac89870fb"} Oct 30 22:00:47.160463 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:47.160059 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71aab5-rt94x" event={"ID":"363ad13f-c717-4b4e-a4fc-f114c21b5d8f","Type":"ContainerStarted","Data":"2b90a086330dbf0249ee10c8692a9c5613f450f4fc903cd529c633f895604358"} Oct 30 22:00:47.216277 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:47.216196 4560 scope.go:122] "RemoveContainer" containerID="ad3bd1addb4ef4b317eff2ced225418bb7f14f064bee5dfead2c5a43fcb60987" Oct 30 22:00:47.280011 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:47.279969 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71aab5-rt94x" podStartSLOduration=2.279955077 podStartE2EDuration="2.279955077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:47.277669882 +0000 UTC m=+2822.832369660" watchObservedRunningTime="2026-10-30 22:00:47.279955077 +0000 UTC m=+2822.834654853" Oct 30 22:00:47.321402 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:47.321382 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-kfkwt" Oct 30 22:00:47.397219 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:47.397193 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d5d0829-37d1-4139-8c40-6c7cc3ba92b1-kube-api-access-nrqwc\" (UniqueName: \"kubernetes.io/projected/6d5d0829-37d1-4139-8c40-6c7cc3ba92b1-kube-api-access-nrqwc\") pod \"6d5d0829-37d1-4139-8c40-6c7cc3ba92b1\" (UID: \"6d5d0829-37d1-4139-8c40-6c7cc3ba92b1\") " Oct 30 22:00:47.397406 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:47.397296 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d5d0829-37d1-4139-8c40-6c7cc3ba92b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d5d0829-37d1-4139-8c40-6c7cc3ba92b1-maas-api-openshift-ca\") pod \"6d5d0829-37d1-4139-8c40-6c7cc3ba92b1\" (UID: \"6d5d0829-37d1-4139-8c40-6c7cc3ba92b1\") " Oct 30 22:00:47.397406 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:47.397368 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d5d0829-37d1-4139-8c40-6c7cc3ba92b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d5d0829-37d1-4139-8c40-6c7cc3ba92b1-maas-api-xks-ca\") pod \"6d5d0829-37d1-4139-8c40-6c7cc3ba92b1\" (UID: \"6d5d0829-37d1-4139-8c40-6c7cc3ba92b1\") " Oct 30 22:00:47.397640 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:47.397613 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d5d0829-37d1-4139-8c40-6c7cc3ba92b1-maas-api-openshift-ca" pod "6d5d0829-37d1-4139-8c40-6c7cc3ba92b1" (UID: "6d5d0829-37d1-4139-8c40-6c7cc3ba92b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:47.397859 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:47.397840 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d5d0829-37d1-4139-8c40-6c7cc3ba92b1-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:47.399365 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:47.399329 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d5d0829-37d1-4139-8c40-6c7cc3ba92b1-kube-api-access-nrqwc" pod "6d5d0829-37d1-4139-8c40-6c7cc3ba92b1" (UID: "6d5d0829-37d1-4139-8c40-6c7cc3ba92b1"). InnerVolumeSpecName "kube-api-access-nrqwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:47.399473 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:47.399459 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d5d0829-37d1-4139-8c40-6c7cc3ba92b1-maas-api-xks-ca" pod "6d5d0829-37d1-4139-8c40-6c7cc3ba92b1" (UID: "6d5d0829-37d1-4139-8c40-6c7cc3ba92b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:00:47.498811 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:47.498778 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrqwc\" (UniqueName: \"kubernetes.io/projected/6d5d0829-37d1-4139-8c40-6c7cc3ba92b1-kube-api-access-nrqwc\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:47.498811 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:47.498809 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d5d0829-37d1-4139-8c40-6c7cc3ba92b1-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:48.168407 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:48.168369 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-kfkwt" Oct 30 22:00:48.168887 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:48.168371 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-kfkwt" event={"ID":"6d5d0829-37d1-4139-8c40-6c7cc3ba92b1","Type":"ContainerDied","Data":"a5c305ac25ea1ced5985b7a91f45482c139f0b6c3cd0bdd2686ed40839080a41"} Oct 30 22:00:48.168887 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:48.168508 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5c305ac25ea1ced5985b7a91f45482c139f0b6c3cd0bdd2686ed40839080a41" Oct 30 22:00:48.170290 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:48.170267 4560 generic.go:356] "Generic (PLEG): container finished" podID="363ad13f-c717-4b4e-a4fc-f114c21b5d8f" containerID="a445c0223c04cd4fb73dda7a38014e56f31062b45ea7ecb33faa691ac89870fb" exitCode=0 Oct 30 22:00:48.170440 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:48.170365 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71aab5-rt94x" event={"ID":"363ad13f-c717-4b4e-a4fc-f114c21b5d8f","Type":"ContainerDied","Data":"a445c0223c04cd4fb73dda7a38014e56f31062b45ea7ecb33faa691ac89870fb"} Oct 30 22:00:48.616759 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:48.616710 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" Oct 30 22:00:48.709405 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:48.709378 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-credential-socket\") pod \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\" (UID: \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\") " Oct 30 22:00:48.709567 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:48.709418 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-workload-socket\") pod \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\" (UID: \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\") " Oct 30 22:00:48.709567 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:48.709445 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-workload-certs\") pod \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\" (UID: \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\") " Oct 30 22:00:48.709567 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:48.709471 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-istiod-ca-cert\") pod \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\" (UID: \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\") " Oct 30 22:00:48.709567 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:48.709549 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-istio-podinfo\") pod \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\" (UID: \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\") " Oct 30 22:00:48.709567 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:48.709568 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-istio-data\") pod \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\" (UID: \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\") " Oct 30 22:00:48.709849 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:48.709614 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-istio-envoy\") pod \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\" (UID: \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\") " Oct 30 22:00:48.709849 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:48.709657 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-kube-api-access-khqbx\" (UniqueName: \"kubernetes.io/projected/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-kube-api-access-khqbx\") pod \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\" (UID: \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\") " Oct 30 22:00:48.709849 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:48.709680 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-credential-socket" pod "8e1ac4c8-c9d4-4419-8a37-cabedad5ad01" (UID: "8e1ac4c8-c9d4-4419-8a37-cabedad5ad01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:48.709849 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:48.709721 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-workload-socket" pod "8e1ac4c8-c9d4-4419-8a37-cabedad5ad01" (UID: "8e1ac4c8-c9d4-4419-8a37-cabedad5ad01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:48.709849 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:48.709727 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-istio-token\" (UniqueName: \"kubernetes.io/projected/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-istio-token\") pod \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\" (UID: \"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01\") " Oct 30 22:00:48.709849 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:48.709781 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-workload-certs" pod "8e1ac4c8-c9d4-4419-8a37-cabedad5ad01" (UID: "8e1ac4c8-c9d4-4419-8a37-cabedad5ad01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:48.710166 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:48.709878 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-istiod-ca-cert" pod "8e1ac4c8-c9d4-4419-8a37-cabedad5ad01" (UID: "8e1ac4c8-c9d4-4419-8a37-cabedad5ad01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:48.710218 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:48.710158 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-istio-data" pod "8e1ac4c8-c9d4-4419-8a37-cabedad5ad01" (UID: "8e1ac4c8-c9d4-4419-8a37-cabedad5ad01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:48.710298 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:48.710279 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-istio-data\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:48.710377 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:48.710300 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-credential-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:48.710377 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:48.710334 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-workload-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:48.710377 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:48.710375 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-workload-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:48.710520 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:48.710390 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-istiod-ca-cert\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:48.712163 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:48.712140 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-istio-envoy" pod "8e1ac4c8-c9d4-4419-8a37-cabedad5ad01" (UID: "8e1ac4c8-c9d4-4419-8a37-cabedad5ad01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:48.712163 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:48.712150 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-istio-podinfo" pod "8e1ac4c8-c9d4-4419-8a37-cabedad5ad01" (UID: "8e1ac4c8-c9d4-4419-8a37-cabedad5ad01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:00:48.712324 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:48.712237 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-kube-api-access-khqbx" pod "8e1ac4c8-c9d4-4419-8a37-cabedad5ad01" (UID: "8e1ac4c8-c9d4-4419-8a37-cabedad5ad01"). InnerVolumeSpecName "kube-api-access-khqbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:48.712324 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:48.712261 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-istio-token" pod "8e1ac4c8-c9d4-4419-8a37-cabedad5ad01" (UID: "8e1ac4c8-c9d4-4419-8a37-cabedad5ad01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:48.812192 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:48.812155 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-istio-podinfo\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:48.812192 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:48.812189 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-istio-envoy\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:48.812461 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:48.812206 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khqbx\" (UniqueName: \"kubernetes.io/projected/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-kube-api-access-khqbx\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:48.812461 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:48.812219 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01-istio-token\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:49.178058 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:49.178023 4560 generic.go:356] "Generic (PLEG): container finished" podID="8e1ac4c8-c9d4-4419-8a37-cabedad5ad01" containerID="2713788d053dd37274192bceae58790d357dc83ecdb7f696140b4ebded3d85da" exitCode=0 Oct 30 22:00:49.178565 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:49.178076 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" Oct 30 22:00:49.178565 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:49.178106 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" event={"ID":"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01","Type":"ContainerDied","Data":"2713788d053dd37274192bceae58790d357dc83ecdb7f696140b4ebded3d85da"} Oct 30 22:00:49.178565 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:49.178159 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b" event={"ID":"8e1ac4c8-c9d4-4419-8a37-cabedad5ad01","Type":"ContainerDied","Data":"3c1d9964e501511a4953c70a9f94d4d4a17c59e1bd9484db90ced20fc393298f"} Oct 30 22:00:49.178565 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:49.178183 4560 scope.go:122] "RemoveContainer" containerID="2713788d053dd37274192bceae58790d357dc83ecdb7f696140b4ebded3d85da" Oct 30 22:00:49.189330 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:49.189032 4560 scope.go:122] "RemoveContainer" containerID="2713788d053dd37274192bceae58790d357dc83ecdb7f696140b4ebded3d85da" Oct 30 22:00:49.189472 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:00:49.189328 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2713788d053dd37274192bceae58790d357dc83ecdb7f696140b4ebded3d85da\": container with ID starting with 2713788d053dd37274192bceae58790d357dc83ecdb7f696140b4ebded3d85da not found: ID does not exist" containerID="2713788d053dd37274192bceae58790d357dc83ecdb7f696140b4ebded3d85da" Oct 30 22:00:49.189472 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:49.189383 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2713788d053dd37274192bceae58790d357dc83ecdb7f696140b4ebded3d85da"} err="failed to get container status \"2713788d053dd37274192bceae58790d357dc83ecdb7f696140b4ebded3d85da\": rpc error: code = NotFound desc = could not find container \"2713788d053dd37274192bceae58790d357dc83ecdb7f696140b4ebded3d85da\": container with ID starting with 2713788d053dd37274192bceae58790d357dc83ecdb7f696140b4ebded3d85da not found: ID does not exist" Oct 30 22:00:49.329805 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:49.329784 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71aab5-rt94x" Oct 30 22:00:49.353559 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:49.353537 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b"] Oct 30 22:00:49.400672 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:49.400648 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-554f46-openshift-default-79c6b6dcf7-2629b"] Oct 30 22:00:49.419610 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:49.419587 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/363ad13f-c717-4b4e-a4fc-f114c21b5d8f-kube-api-access-7wbdm\" (UniqueName: \"kubernetes.io/projected/363ad13f-c717-4b4e-a4fc-f114c21b5d8f-kube-api-access-7wbdm\") pod \"363ad13f-c717-4b4e-a4fc-f114c21b5d8f\" (UID: \"363ad13f-c717-4b4e-a4fc-f114c21b5d8f\") " Oct 30 22:00:49.419716 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:49.419701 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/363ad13f-c717-4b4e-a4fc-f114c21b5d8f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363ad13f-c717-4b4e-a4fc-f114c21b5d8f-maas-api-xks-ca\") pod \"363ad13f-c717-4b4e-a4fc-f114c21b5d8f\" (UID: \"363ad13f-c717-4b4e-a4fc-f114c21b5d8f\") " Oct 30 22:00:49.419772 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:49.419729 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/363ad13f-c717-4b4e-a4fc-f114c21b5d8f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363ad13f-c717-4b4e-a4fc-f114c21b5d8f-maas-api-openshift-ca\") pod \"363ad13f-c717-4b4e-a4fc-f114c21b5d8f\" (UID: \"363ad13f-c717-4b4e-a4fc-f114c21b5d8f\") " Oct 30 22:00:49.420134 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:49.420094 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/363ad13f-c717-4b4e-a4fc-f114c21b5d8f-maas-api-openshift-ca" pod "363ad13f-c717-4b4e-a4fc-f114c21b5d8f" (UID: "363ad13f-c717-4b4e-a4fc-f114c21b5d8f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:49.420282 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:49.420264 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363ad13f-c717-4b4e-a4fc-f114c21b5d8f-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:49.421664 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:49.421637 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/363ad13f-c717-4b4e-a4fc-f114c21b5d8f-kube-api-access-7wbdm" pod "363ad13f-c717-4b4e-a4fc-f114c21b5d8f" (UID: "363ad13f-c717-4b4e-a4fc-f114c21b5d8f"). InnerVolumeSpecName "kube-api-access-7wbdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:49.421735 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:49.421706 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/363ad13f-c717-4b4e-a4fc-f114c21b5d8f-maas-api-xks-ca" pod "363ad13f-c717-4b4e-a4fc-f114c21b5d8f" (UID: "363ad13f-c717-4b4e-a4fc-f114c21b5d8f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:00:49.521122 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:49.521090 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wbdm\" (UniqueName: \"kubernetes.io/projected/363ad13f-c717-4b4e-a4fc-f114c21b5d8f-kube-api-access-7wbdm\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:49.521122 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:49.521119 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363ad13f-c717-4b4e-a4fc-f114c21b5d8f-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:49.684858 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:00:49.684777 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:00:50.185451 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:50.185425 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71aab5-rt94x" Oct 30 22:00:50.185859 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:50.185459 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71aab5-rt94x" event={"ID":"363ad13f-c717-4b4e-a4fc-f114c21b5d8f","Type":"ContainerDied","Data":"2b90a086330dbf0249ee10c8692a9c5613f450f4fc903cd529c633f895604358"} Oct 30 22:00:50.185859 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:50.185501 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b90a086330dbf0249ee10c8692a9c5613f450f4fc903cd529c633f895604358" Oct 30 22:00:50.577295 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:50.577255 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e1ac4c8-c9d4-4419-8a37-cabedad5ad01" path="/var/lib/kubelet/pods/8e1ac4c8-c9d4-4419-8a37-cabedad5ad01/volumes" Oct 30 22:00:52.544417 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:52.544385 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2df7b6-n2f5r"] Oct 30 22:00:52.590837 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:52.590807 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2df7b6-n2f5r"] Oct 30 22:00:54.575520 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:54.575487 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9993cae3-8e92-479a-b182-0fe7be890338" path="/var/lib/kubelet/pods/9993cae3-8e92-479a-b182-0fe7be890338/volumes" Oct 30 22:00:54.881551 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:54.881468 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bfdf30b9-66766fc856-gscj4"] Oct 30 22:00:54.983728 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:54.983694 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bfdf30b9-6c99dcd5f5-sp9kv"] Oct 30 22:00:59.211155 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:59.211065 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71c0c7-ptgs7"] Oct 30 22:00:59.257006 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:59.256979 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71c0c7-ptgs7"] Oct 30 22:00:59.882472 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:59.882418 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-bfdf30b9-66766fc856-gscj4" podUID="69c4f942-a6f7-4d0f-9f30-10a1de67e587" containerName="payload-processing" containerID="cri-o://d262adbd4f8258d45eb2c9a70adb0b526a9e9e2faf427d68cdd0f53b45afbf9f" gracePeriod=25 Oct 30 22:00:59.984642 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:00:59.984584 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-bfdf30b9-6c99dcd5f5-sp9kv" podUID="c6372f6e-8f23-43b4-8fbf-33f0fb5f61ac" containerName="payload-pre-processing" containerID="cri-o://0e2866238e4ababe75c029fa69d4bf0ffd3c778ab6b45644775f29138d52478d" gracePeriod=25 Oct 30 22:01:00.160795 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:00.160455 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bfdf30b9-66766fc856-gscj4" Oct 30 22:01:00.220880 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:00.220833 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69c4f942-a6f7-4d0f-9f30-10a1de67e587-kube-api-access-45zsx\" (UniqueName: \"kubernetes.io/projected/69c4f942-a6f7-4d0f-9f30-10a1de67e587-kube-api-access-45zsx\") pod \"69c4f942-a6f7-4d0f-9f30-10a1de67e587\" (UID: \"69c4f942-a6f7-4d0f-9f30-10a1de67e587\") " Oct 30 22:01:00.223829 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:00.223795 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69c4f942-a6f7-4d0f-9f30-10a1de67e587-kube-api-access-45zsx" pod "69c4f942-a6f7-4d0f-9f30-10a1de67e587" (UID: "69c4f942-a6f7-4d0f-9f30-10a1de67e587"). InnerVolumeSpecName "kube-api-access-45zsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:00.247173 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:00.247151 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bfdf30b9-6c99dcd5f5-sp9kv" Oct 30 22:01:00.294845 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:00.294794 4560 generic.go:356] "Generic (PLEG): container finished" podID="69c4f942-a6f7-4d0f-9f30-10a1de67e587" containerID="d262adbd4f8258d45eb2c9a70adb0b526a9e9e2faf427d68cdd0f53b45afbf9f" exitCode=0 Oct 30 22:01:00.297007 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:00.294846 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bfdf30b9-66766fc856-gscj4" Oct 30 22:01:00.297226 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:00.294867 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bfdf30b9-66766fc856-gscj4" event={"ID":"69c4f942-a6f7-4d0f-9f30-10a1de67e587","Type":"ContainerDied","Data":"d262adbd4f8258d45eb2c9a70adb0b526a9e9e2faf427d68cdd0f53b45afbf9f"} Oct 30 22:01:00.297318 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:00.297298 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bfdf30b9-66766fc856-gscj4" event={"ID":"69c4f942-a6f7-4d0f-9f30-10a1de67e587","Type":"ContainerDied","Data":"a0da52d466950589a3d1c28d1717042a194a8b295a83dd5fee168437af1c98f6"} Oct 30 22:01:00.297414 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:00.297351 4560 scope.go:122] "RemoveContainer" containerID="d262adbd4f8258d45eb2c9a70adb0b526a9e9e2faf427d68cdd0f53b45afbf9f" Oct 30 22:01:00.307302 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:00.307274 4560 generic.go:356] "Generic (PLEG): container finished" podID="c6372f6e-8f23-43b4-8fbf-33f0fb5f61ac" containerID="0e2866238e4ababe75c029fa69d4bf0ffd3c778ab6b45644775f29138d52478d" exitCode=0 Oct 30 22:01:00.307626 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:00.307516 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bfdf30b9-6c99dcd5f5-sp9kv" event={"ID":"c6372f6e-8f23-43b4-8fbf-33f0fb5f61ac","Type":"ContainerDied","Data":"0e2866238e4ababe75c029fa69d4bf0ffd3c778ab6b45644775f29138d52478d"} Oct 30 22:01:00.307880 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:00.307856 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bfdf30b9-6c99dcd5f5-sp9kv" event={"ID":"c6372f6e-8f23-43b4-8fbf-33f0fb5f61ac","Type":"ContainerDied","Data":"1799db2f5faf5ec17c7b0a2466a3fa94d5e5e0b405dc15ef48c3437dae1565d4"} Oct 30 22:01:00.308065 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:00.308046 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bfdf30b9-6c99dcd5f5-sp9kv" Oct 30 22:01:00.320969 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:00.320601 4560 scope.go:122] "RemoveContainer" containerID="d262adbd4f8258d45eb2c9a70adb0b526a9e9e2faf427d68cdd0f53b45afbf9f" Oct 30 22:01:00.320969 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:01:00.320938 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d262adbd4f8258d45eb2c9a70adb0b526a9e9e2faf427d68cdd0f53b45afbf9f\": container with ID starting with d262adbd4f8258d45eb2c9a70adb0b526a9e9e2faf427d68cdd0f53b45afbf9f not found: ID does not exist" containerID="d262adbd4f8258d45eb2c9a70adb0b526a9e9e2faf427d68cdd0f53b45afbf9f" Oct 30 22:01:00.321051 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:00.320964 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d262adbd4f8258d45eb2c9a70adb0b526a9e9e2faf427d68cdd0f53b45afbf9f"} err="failed to get container status \"d262adbd4f8258d45eb2c9a70adb0b526a9e9e2faf427d68cdd0f53b45afbf9f\": rpc error: code = NotFound desc = could not find container \"d262adbd4f8258d45eb2c9a70adb0b526a9e9e2faf427d68cdd0f53b45afbf9f\": container with ID starting with d262adbd4f8258d45eb2c9a70adb0b526a9e9e2faf427d68cdd0f53b45afbf9f not found: ID does not exist" Oct 30 22:01:00.321051 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:00.320983 4560 scope.go:122] "RemoveContainer" containerID="0e2866238e4ababe75c029fa69d4bf0ffd3c778ab6b45644775f29138d52478d" Oct 30 22:01:00.321745 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:00.321726 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69c4f942-a6f7-4d0f-9f30-10a1de67e587-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69c4f942-a6f7-4d0f-9f30-10a1de67e587-plugins-config-volume\") pod \"69c4f942-a6f7-4d0f-9f30-10a1de67e587\" (UID: \"69c4f942-a6f7-4d0f-9f30-10a1de67e587\") " Oct 30 22:01:00.321869 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:00.321839 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6372f6e-8f23-43b4-8fbf-33f0fb5f61ac-kube-api-access-2q9n5\" (UniqueName: \"kubernetes.io/projected/c6372f6e-8f23-43b4-8fbf-33f0fb5f61ac-kube-api-access-2q9n5\") pod \"c6372f6e-8f23-43b4-8fbf-33f0fb5f61ac\" (UID: \"c6372f6e-8f23-43b4-8fbf-33f0fb5f61ac\") " Oct 30 22:01:00.322230 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:00.321889 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6372f6e-8f23-43b4-8fbf-33f0fb5f61ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6372f6e-8f23-43b4-8fbf-33f0fb5f61ac-plugins-config-volume\") pod \"c6372f6e-8f23-43b4-8fbf-33f0fb5f61ac\" (UID: \"c6372f6e-8f23-43b4-8fbf-33f0fb5f61ac\") " Oct 30 22:01:00.322230 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:00.322103 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69c4f942-a6f7-4d0f-9f30-10a1de67e587-plugins-config-volume" pod "69c4f942-a6f7-4d0f-9f30-10a1de67e587" (UID: "69c4f942-a6f7-4d0f-9f30-10a1de67e587"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:01:00.322440 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:00.322392 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6372f6e-8f23-43b4-8fbf-33f0fb5f61ac-plugins-config-volume" pod "c6372f6e-8f23-43b4-8fbf-33f0fb5f61ac" (UID: "c6372f6e-8f23-43b4-8fbf-33f0fb5f61ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:01:00.322556 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:00.322536 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69c4f942-a6f7-4d0f-9f30-10a1de67e587-plugins-config-volume\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:00.322608 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:00.322561 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6372f6e-8f23-43b4-8fbf-33f0fb5f61ac-plugins-config-volume\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:00.322608 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:00.322575 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45zsx\" (UniqueName: \"kubernetes.io/projected/69c4f942-a6f7-4d0f-9f30-10a1de67e587-kube-api-access-45zsx\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:00.324809 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:00.324786 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6372f6e-8f23-43b4-8fbf-33f0fb5f61ac-kube-api-access-2q9n5" pod "c6372f6e-8f23-43b4-8fbf-33f0fb5f61ac" (UID: "c6372f6e-8f23-43b4-8fbf-33f0fb5f61ac"). InnerVolumeSpecName "kube-api-access-2q9n5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:00.332023 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:00.332004 4560 scope.go:122] "RemoveContainer" containerID="0e2866238e4ababe75c029fa69d4bf0ffd3c778ab6b45644775f29138d52478d" Oct 30 22:01:00.332355 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:01:00.332309 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e2866238e4ababe75c029fa69d4bf0ffd3c778ab6b45644775f29138d52478d\": container with ID starting with 0e2866238e4ababe75c029fa69d4bf0ffd3c778ab6b45644775f29138d52478d not found: ID does not exist" containerID="0e2866238e4ababe75c029fa69d4bf0ffd3c778ab6b45644775f29138d52478d" Oct 30 22:01:00.332438 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:00.332363 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e2866238e4ababe75c029fa69d4bf0ffd3c778ab6b45644775f29138d52478d"} err="failed to get container status \"0e2866238e4ababe75c029fa69d4bf0ffd3c778ab6b45644775f29138d52478d\": rpc error: code = NotFound desc = could not find container \"0e2866238e4ababe75c029fa69d4bf0ffd3c778ab6b45644775f29138d52478d\": container with ID starting with 0e2866238e4ababe75c029fa69d4bf0ffd3c778ab6b45644775f29138d52478d not found: ID does not exist" Oct 30 22:01:00.423789 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:00.423712 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q9n5\" (UniqueName: \"kubernetes.io/projected/c6372f6e-8f23-43b4-8fbf-33f0fb5f61ac-kube-api-access-2q9n5\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:00.575699 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:00.575668 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7b824c4-575b-4c1a-bcf4-0f833dd21f70" path="/var/lib/kubelet/pods/d7b824c4-575b-4c1a-bcf4-0f833dd21f70/volumes" Oct 30 22:01:00.654431 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:00.654406 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-40bb78-6qpv8"] Oct 30 22:01:00.658196 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:00.658174 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-40bb78-6qpv8"] Oct 30 22:01:00.708532 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:00.708477 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bfdf30b9-66766fc856-gscj4"] Oct 30 22:01:00.712868 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:00.712848 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bfdf30b9-66766fc856-gscj4"] Oct 30 22:01:00.817962 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:00.817932 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bfdf30b9-6c99dcd5f5-sp9kv"] Oct 30 22:01:00.824409 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:00.824386 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bfdf30b9-6c99dcd5f5-sp9kv"] Oct 30 22:01:02.514979 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:02.514945 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19dd4f-tg7k6"] Oct 30 22:01:02.515442 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:02.515427 4560 cpu_manager.go:414] "containerMap: removing container" podUID="6d5d0829-37d1-4139-8c40-6c7cc3ba92b1" containerName="revoke-keys" Oct 30 22:01:02.515493 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:02.515445 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d5d0829-37d1-4139-8c40-6c7cc3ba92b1" containerName="revoke-keys" Oct 30 22:01:02.515493 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:02.515461 4560 cpu_manager.go:414] "containerMap: removing container" podUID="c6372f6e-8f23-43b4-8fbf-33f0fb5f61ac" containerName="payload-pre-processing" Oct 30 22:01:02.515493 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:02.515469 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6372f6e-8f23-43b4-8fbf-33f0fb5f61ac" containerName="payload-pre-processing" Oct 30 22:01:02.515646 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:02.515495 4560 cpu_manager.go:414] "containerMap: removing container" podUID="69c4f942-a6f7-4d0f-9f30-10a1de67e587" containerName="payload-processing" Oct 30 22:01:02.515646 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:02.515500 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69c4f942-a6f7-4d0f-9f30-10a1de67e587" containerName="payload-processing" Oct 30 22:01:02.515646 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:02.515514 4560 cpu_manager.go:414] "containerMap: removing container" podUID="8e1ac4c8-c9d4-4419-8a37-cabedad5ad01" containerName="istio-proxy" Oct 30 22:01:02.515646 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:02.515518 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e1ac4c8-c9d4-4419-8a37-cabedad5ad01" containerName="istio-proxy" Oct 30 22:01:02.515646 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:02.515529 4560 cpu_manager.go:414] "containerMap: removing container" podUID="363ad13f-c717-4b4e-a4fc-f114c21b5d8f" containerName="revoke-keys" Oct 30 22:01:02.515646 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:02.515537 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="363ad13f-c717-4b4e-a4fc-f114c21b5d8f" containerName="revoke-keys" Oct 30 22:01:02.515646 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:02.515606 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e1ac4c8-c9d4-4419-8a37-cabedad5ad01" containerName="istio-proxy" Oct 30 22:01:02.515646 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:02.515622 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="363ad13f-c717-4b4e-a4fc-f114c21b5d8f" containerName="revoke-keys" Oct 30 22:01:02.515646 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:02.515629 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d5d0829-37d1-4139-8c40-6c7cc3ba92b1" containerName="revoke-keys" Oct 30 22:01:02.515646 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:02.515634 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="69c4f942-a6f7-4d0f-9f30-10a1de67e587" containerName="payload-processing" Oct 30 22:01:02.515646 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:02.515640 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6372f6e-8f23-43b4-8fbf-33f0fb5f61ac" containerName="payload-pre-processing" Oct 30 22:01:02.520484 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:02.520465 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19dd4f-tg7k6" Oct 30 22:01:02.554227 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:02.554187 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c965fda1-f5b4-402a-99c7-f9ec481edabd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-19dd4f-tg7k6\" (UID: \"c965fda1-f5b4-402a-99c7-f9ec481edabd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19dd4f-tg7k6" Oct 30 22:01:02.554401 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:02.554241 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c965fda1-f5b4-402a-99c7-f9ec481edabd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-19dd4f-tg7k6\" (UID: \"c965fda1-f5b4-402a-99c7-f9ec481edabd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19dd4f-tg7k6" Oct 30 22:01:02.554401 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:02.554359 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf82c\" (UniqueName: \"kubernetes.io/projected/c965fda1-f5b4-402a-99c7-f9ec481edabd-kube-api-access-cf82c\") pod \"maas-api-revoke-keys-e2e-worker-w1-19dd4f-tg7k6\" (UID: \"c965fda1-f5b4-402a-99c7-f9ec481edabd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19dd4f-tg7k6" Oct 30 22:01:02.577965 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:02.577905 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4632ebfb-a5e6-4c7f-9407-8dd343358cf2" path="/var/lib/kubelet/pods/4632ebfb-a5e6-4c7f-9407-8dd343358cf2/volumes" Oct 30 22:01:02.578606 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:02.578581 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69c4f942-a6f7-4d0f-9f30-10a1de67e587" path="/var/lib/kubelet/pods/69c4f942-a6f7-4d0f-9f30-10a1de67e587/volumes" Oct 30 22:01:02.579016 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:02.578958 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6372f6e-8f23-43b4-8fbf-33f0fb5f61ac" path="/var/lib/kubelet/pods/c6372f6e-8f23-43b4-8fbf-33f0fb5f61ac/volumes" Oct 30 22:01:02.579280 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:02.579263 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19dd4f-tg7k6"] Oct 30 22:01:02.620264 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:02.620227 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:01:02.620264 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:02.620244 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:01:02.620264 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:02.620257 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:01:02.620516 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:02.620258 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:01:02.665335 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:02.665283 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c965fda1-f5b4-402a-99c7-f9ec481edabd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-19dd4f-tg7k6\" (UID: \"c965fda1-f5b4-402a-99c7-f9ec481edabd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19dd4f-tg7k6" Oct 30 22:01:02.665562 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:02.665392 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c965fda1-f5b4-402a-99c7-f9ec481edabd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-19dd4f-tg7k6\" (UID: \"c965fda1-f5b4-402a-99c7-f9ec481edabd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19dd4f-tg7k6" Oct 30 22:01:02.665562 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:02.665479 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf82c\" (UniqueName: \"kubernetes.io/projected/c965fda1-f5b4-402a-99c7-f9ec481edabd-kube-api-access-cf82c\") pod \"maas-api-revoke-keys-e2e-worker-w1-19dd4f-tg7k6\" (UID: \"c965fda1-f5b4-402a-99c7-f9ec481edabd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19dd4f-tg7k6" Oct 30 22:01:02.666069 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:02.666049 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c965fda1-f5b4-402a-99c7-f9ec481edabd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-19dd4f-tg7k6\" (UID: \"c965fda1-f5b4-402a-99c7-f9ec481edabd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19dd4f-tg7k6" Oct 30 22:01:02.667688 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:02.667669 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c965fda1-f5b4-402a-99c7-f9ec481edabd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-19dd4f-tg7k6\" (UID: \"c965fda1-f5b4-402a-99c7-f9ec481edabd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19dd4f-tg7k6" Oct 30 22:01:02.723687 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:02.723652 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf82c\" (UniqueName: \"kubernetes.io/projected/c965fda1-f5b4-402a-99c7-f9ec481edabd-kube-api-access-cf82c\") pod \"maas-api-revoke-keys-e2e-worker-w1-19dd4f-tg7k6\" (UID: \"c965fda1-f5b4-402a-99c7-f9ec481edabd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19dd4f-tg7k6" Oct 30 22:01:02.835452 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:02.835375 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19dd4f-tg7k6" Oct 30 22:01:03.233072 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:03.233034 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19dd4f-tg7k6"] Oct 30 22:01:03.250999 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:01:03.250968 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc965fda1_f5b4_402a_99c7_f9ec481edabd.slice/crio-a0be2d2344f4957554e0604596eff0a61f9e71074aeed726a8e470d12df0b08f WatchSource:0}: Error finding container a0be2d2344f4957554e0604596eff0a61f9e71074aeed726a8e470d12df0b08f: Status 404 returned error can't find the container with id a0be2d2344f4957554e0604596eff0a61f9e71074aeed726a8e470d12df0b08f Oct 30 22:01:03.328760 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:03.328721 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19dd4f-tg7k6" event={"ID":"c965fda1-f5b4-402a-99c7-f9ec481edabd","Type":"ContainerStarted","Data":"a0be2d2344f4957554e0604596eff0a61f9e71074aeed726a8e470d12df0b08f"} Oct 30 22:01:04.336245 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:04.336205 4560 generic.go:356] "Generic (PLEG): container finished" podID="c965fda1-f5b4-402a-99c7-f9ec481edabd" containerID="8a2e70151d8c4cb0236662dfbc6b5a628eeeb7c4f9efe9bc5b749ac962a023c4" exitCode=0 Oct 30 22:01:04.336657 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:04.336245 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19dd4f-tg7k6" event={"ID":"c965fda1-f5b4-402a-99c7-f9ec481edabd","Type":"ContainerDied","Data":"8a2e70151d8c4cb0236662dfbc6b5a628eeeb7c4f9efe9bc5b749ac962a023c4"} Oct 30 22:01:05.501212 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:05.501187 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19dd4f-tg7k6" Oct 30 22:01:05.603079 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:05.603046 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c965fda1-f5b4-402a-99c7-f9ec481edabd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c965fda1-f5b4-402a-99c7-f9ec481edabd-maas-api-openshift-ca\") pod \"c965fda1-f5b4-402a-99c7-f9ec481edabd\" (UID: \"c965fda1-f5b4-402a-99c7-f9ec481edabd\") " Oct 30 22:01:05.603251 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:05.603133 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c965fda1-f5b4-402a-99c7-f9ec481edabd-kube-api-access-cf82c\" (UniqueName: \"kubernetes.io/projected/c965fda1-f5b4-402a-99c7-f9ec481edabd-kube-api-access-cf82c\") pod \"c965fda1-f5b4-402a-99c7-f9ec481edabd\" (UID: \"c965fda1-f5b4-402a-99c7-f9ec481edabd\") " Oct 30 22:01:05.603251 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:05.603193 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c965fda1-f5b4-402a-99c7-f9ec481edabd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c965fda1-f5b4-402a-99c7-f9ec481edabd-maas-api-xks-ca\") pod \"c965fda1-f5b4-402a-99c7-f9ec481edabd\" (UID: \"c965fda1-f5b4-402a-99c7-f9ec481edabd\") " Oct 30 22:01:05.603418 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:05.603370 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c965fda1-f5b4-402a-99c7-f9ec481edabd-maas-api-openshift-ca" pod "c965fda1-f5b4-402a-99c7-f9ec481edabd" (UID: "c965fda1-f5b4-402a-99c7-f9ec481edabd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:01:05.603620 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:05.603599 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c965fda1-f5b4-402a-99c7-f9ec481edabd-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:05.605226 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:05.605196 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c965fda1-f5b4-402a-99c7-f9ec481edabd-kube-api-access-cf82c" pod "c965fda1-f5b4-402a-99c7-f9ec481edabd" (UID: "c965fda1-f5b4-402a-99c7-f9ec481edabd"). InnerVolumeSpecName "kube-api-access-cf82c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:05.605312 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:05.605232 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c965fda1-f5b4-402a-99c7-f9ec481edabd-maas-api-xks-ca" pod "c965fda1-f5b4-402a-99c7-f9ec481edabd" (UID: "c965fda1-f5b4-402a-99c7-f9ec481edabd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:01:05.726364 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:05.725452 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf82c\" (UniqueName: \"kubernetes.io/projected/c965fda1-f5b4-402a-99c7-f9ec481edabd-kube-api-access-cf82c\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:05.726364 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:05.725491 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c965fda1-f5b4-402a-99c7-f9ec481edabd-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:06.364129 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:06.364078 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19dd4f-tg7k6" event={"ID":"c965fda1-f5b4-402a-99c7-f9ec481edabd","Type":"ContainerDied","Data":"a0be2d2344f4957554e0604596eff0a61f9e71074aeed726a8e470d12df0b08f"} Oct 30 22:01:06.364129 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:06.364118 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19dd4f-tg7k6" Oct 30 22:01:06.364400 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:06.364213 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0be2d2344f4957554e0604596eff0a61f9e71074aeed726a8e470d12df0b08f" Oct 30 22:01:06.848674 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:06.848643 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5"] Oct 30 22:01:06.868278 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:06.868249 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm"] Oct 30 22:01:08.407376 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:08.407311 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4e874-284qz"] Oct 30 22:01:08.407893 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:08.407875 4560 cpu_manager.go:414] "containerMap: removing container" podUID="c965fda1-f5b4-402a-99c7-f9ec481edabd" containerName="revoke-keys" Oct 30 22:01:08.407946 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:08.407895 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c965fda1-f5b4-402a-99c7-f9ec481edabd" containerName="revoke-keys" Oct 30 22:01:08.407997 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:08.407989 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="c965fda1-f5b4-402a-99c7-f9ec481edabd" containerName="revoke-keys" Oct 30 22:01:08.411321 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:08.411304 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4e874-284qz" Oct 30 22:01:08.466679 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:08.466643 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4e874-284qz"] Oct 30 22:01:08.510035 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:08.510006 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:01:08.510226 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:08.510208 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:01:08.510315 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:08.510234 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:01:08.510379 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:08.510364 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:01:08.567923 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:08.567861 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cbcd79a-7e7c-4bca-9659-9bae7235f3d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c4e874-284qz\" (UID: \"2cbcd79a-7e7c-4bca-9659-9bae7235f3d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4e874-284qz" Oct 30 22:01:08.574166 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:08.574128 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cbcd79a-7e7c-4bca-9659-9bae7235f3d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c4e874-284qz\" (UID: \"2cbcd79a-7e7c-4bca-9659-9bae7235f3d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4e874-284qz" Oct 30 22:01:08.574480 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:08.574442 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c82g\" (UniqueName: \"kubernetes.io/projected/2cbcd79a-7e7c-4bca-9659-9bae7235f3d8-kube-api-access-2c82g\") pod \"maas-api-revoke-keys-e2e-api-b-c4e874-284qz\" (UID: \"2cbcd79a-7e7c-4bca-9659-9bae7235f3d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4e874-284qz" Oct 30 22:01:08.675993 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:08.675907 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cbcd79a-7e7c-4bca-9659-9bae7235f3d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c4e874-284qz\" (UID: \"2cbcd79a-7e7c-4bca-9659-9bae7235f3d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4e874-284qz" Oct 30 22:01:08.675993 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:08.675954 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cbcd79a-7e7c-4bca-9659-9bae7235f3d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c4e874-284qz\" (UID: \"2cbcd79a-7e7c-4bca-9659-9bae7235f3d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4e874-284qz" Oct 30 22:01:08.676220 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:08.676063 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c82g\" (UniqueName: \"kubernetes.io/projected/2cbcd79a-7e7c-4bca-9659-9bae7235f3d8-kube-api-access-2c82g\") pod \"maas-api-revoke-keys-e2e-api-b-c4e874-284qz\" (UID: \"2cbcd79a-7e7c-4bca-9659-9bae7235f3d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4e874-284qz" Oct 30 22:01:08.676659 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:08.676635 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cbcd79a-7e7c-4bca-9659-9bae7235f3d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c4e874-284qz\" (UID: \"2cbcd79a-7e7c-4bca-9659-9bae7235f3d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4e874-284qz" Oct 30 22:01:08.678199 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:08.678180 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cbcd79a-7e7c-4bca-9659-9bae7235f3d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c4e874-284qz\" (UID: \"2cbcd79a-7e7c-4bca-9659-9bae7235f3d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4e874-284qz" Oct 30 22:01:08.757457 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:08.757429 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c82g\" (UniqueName: \"kubernetes.io/projected/2cbcd79a-7e7c-4bca-9659-9bae7235f3d8-kube-api-access-2c82g\") pod \"maas-api-revoke-keys-e2e-api-b-c4e874-284qz\" (UID: \"2cbcd79a-7e7c-4bca-9659-9bae7235f3d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4e874-284qz" Oct 30 22:01:09.039359 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:09.039033 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4e874-284qz" Oct 30 22:01:09.521469 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:09.521433 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4e874-284qz"] Oct 30 22:01:09.527171 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:01:09.527142 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cbcd79a_7e7c_4bca_9659_9bae7235f3d8.slice/crio-76b38c09d8be952b140166b9e3b8272e2ff4c5a0a15996d862abb386a55ae405 WatchSource:0}: Error finding container 76b38c09d8be952b140166b9e3b8272e2ff4c5a0a15996d862abb386a55ae405: Status 404 returned error can't find the container with id 76b38c09d8be952b140166b9e3b8272e2ff4c5a0a15996d862abb386a55ae405 Oct 30 22:01:09.727225 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:09.727190 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ca0a55-7ww8s"] Oct 30 22:01:09.732525 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:09.732504 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ca0a55-7ww8s" Oct 30 22:01:09.786582 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:09.786507 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ca0a55-7ww8s"] Oct 30 22:01:09.813817 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:09.813781 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8fxg\" (UniqueName: \"kubernetes.io/projected/f12246a4-1d55-475b-bcdb-5472da93f641-kube-api-access-n8fxg\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-ca0a55-7ww8s\" (UID: \"f12246a4-1d55-475b-bcdb-5472da93f641\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ca0a55-7ww8s" Oct 30 22:01:09.813961 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:09.813859 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f12246a4-1d55-475b-bcdb-5472da93f641-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-ca0a55-7ww8s\" (UID: \"f12246a4-1d55-475b-bcdb-5472da93f641\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ca0a55-7ww8s" Oct 30 22:01:09.814098 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:09.814072 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f12246a4-1d55-475b-bcdb-5472da93f641-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-ca0a55-7ww8s\" (UID: \"f12246a4-1d55-475b-bcdb-5472da93f641\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ca0a55-7ww8s" Oct 30 22:01:09.914878 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:09.914840 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f12246a4-1d55-475b-bcdb-5472da93f641-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-ca0a55-7ww8s\" (UID: \"f12246a4-1d55-475b-bcdb-5472da93f641\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ca0a55-7ww8s" Oct 30 22:01:09.915043 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:09.914922 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8fxg\" (UniqueName: \"kubernetes.io/projected/f12246a4-1d55-475b-bcdb-5472da93f641-kube-api-access-n8fxg\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-ca0a55-7ww8s\" (UID: \"f12246a4-1d55-475b-bcdb-5472da93f641\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ca0a55-7ww8s" Oct 30 22:01:09.915131 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:09.915075 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f12246a4-1d55-475b-bcdb-5472da93f641-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-ca0a55-7ww8s\" (UID: \"f12246a4-1d55-475b-bcdb-5472da93f641\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ca0a55-7ww8s" Oct 30 22:01:09.915700 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:09.915674 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f12246a4-1d55-475b-bcdb-5472da93f641-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-ca0a55-7ww8s\" (UID: \"f12246a4-1d55-475b-bcdb-5472da93f641\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ca0a55-7ww8s" Oct 30 22:01:09.917189 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:09.917167 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f12246a4-1d55-475b-bcdb-5472da93f641-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-ca0a55-7ww8s\" (UID: \"f12246a4-1d55-475b-bcdb-5472da93f641\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ca0a55-7ww8s" Oct 30 22:01:09.973436 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:09.973401 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8fxg\" (UniqueName: \"kubernetes.io/projected/f12246a4-1d55-475b-bcdb-5472da93f641-kube-api-access-n8fxg\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-ca0a55-7ww8s\" (UID: \"f12246a4-1d55-475b-bcdb-5472da93f641\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ca0a55-7ww8s" Oct 30 22:01:10.066323 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:10.066250 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ca0a55-7ww8s" Oct 30 22:01:10.391000 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:10.390950 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4e874-284qz" event={"ID":"2cbcd79a-7e7c-4bca-9659-9bae7235f3d8","Type":"ContainerStarted","Data":"c79dde18f6f83bfc9a1925c70476c80b921c98e8348f598494bebde3746f59d9"} Oct 30 22:01:10.391149 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:10.391055 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4e874-284qz" event={"ID":"2cbcd79a-7e7c-4bca-9659-9bae7235f3d8","Type":"ContainerStarted","Data":"76b38c09d8be952b140166b9e3b8272e2ff4c5a0a15996d862abb386a55ae405"} Oct 30 22:01:10.441194 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:10.441163 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ca0a55-7ww8s"] Oct 30 22:01:10.456417 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:01:10.456360 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf12246a4_1d55_475b_bcdb_5472da93f641.slice/crio-07aec596c5f5b01539c9837007959870f5122dd8c695b0b2fdcee489a9ecff37 WatchSource:0}: Error finding container 07aec596c5f5b01539c9837007959870f5122dd8c695b0b2fdcee489a9ecff37: Status 404 returned error can't find the container with id 07aec596c5f5b01539c9837007959870f5122dd8c695b0b2fdcee489a9ecff37 Oct 30 22:01:10.519690 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:10.518676 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4e874-284qz" podStartSLOduration=2.51865399 podStartE2EDuration="2.51865399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:10.505838086 +0000 UTC m=+2846.060537863" watchObservedRunningTime="2026-10-30 22:01:10.51865399 +0000 UTC m=+2846.073353767" Oct 30 22:01:11.398296 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:11.398265 4560 generic.go:356] "Generic (PLEG): container finished" podID="2cbcd79a-7e7c-4bca-9659-9bae7235f3d8" containerID="c79dde18f6f83bfc9a1925c70476c80b921c98e8348f598494bebde3746f59d9" exitCode=0 Oct 30 22:01:11.398680 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:11.398316 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4e874-284qz" event={"ID":"2cbcd79a-7e7c-4bca-9659-9bae7235f3d8","Type":"ContainerDied","Data":"c79dde18f6f83bfc9a1925c70476c80b921c98e8348f598494bebde3746f59d9"} Oct 30 22:01:11.399886 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:11.399865 4560 generic.go:356] "Generic (PLEG): container finished" podID="f12246a4-1d55-475b-bcdb-5472da93f641" containerID="5b19aec620a7bc0f4ad91c30083fa4d7347e826c288720df4771bea3bc68c32f" exitCode=0 Oct 30 22:01:11.399967 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:11.399926 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ca0a55-7ww8s" event={"ID":"f12246a4-1d55-475b-bcdb-5472da93f641","Type":"ContainerDied","Data":"5b19aec620a7bc0f4ad91c30083fa4d7347e826c288720df4771bea3bc68c32f"} Oct 30 22:01:11.399967 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:11.399958 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ca0a55-7ww8s" event={"ID":"f12246a4-1d55-475b-bcdb-5472da93f641","Type":"ContainerStarted","Data":"07aec596c5f5b01539c9837007959870f5122dd8c695b0b2fdcee489a9ecff37"} Oct 30 22:01:12.658081 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:12.658056 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ca0a55-7ww8s" Oct 30 22:01:12.661226 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:12.661202 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4e874-284qz" Oct 30 22:01:12.753150 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:12.753120 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f12246a4-1d55-475b-bcdb-5472da93f641-kube-api-access-n8fxg\" (UniqueName: \"kubernetes.io/projected/f12246a4-1d55-475b-bcdb-5472da93f641-kube-api-access-n8fxg\") pod \"f12246a4-1d55-475b-bcdb-5472da93f641\" (UID: \"f12246a4-1d55-475b-bcdb-5472da93f641\") " Oct 30 22:01:12.753332 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:12.753195 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cbcd79a-7e7c-4bca-9659-9bae7235f3d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cbcd79a-7e7c-4bca-9659-9bae7235f3d8-maas-api-openshift-ca\") pod \"2cbcd79a-7e7c-4bca-9659-9bae7235f3d8\" (UID: \"2cbcd79a-7e7c-4bca-9659-9bae7235f3d8\") " Oct 30 22:01:12.753332 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:12.753229 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cbcd79a-7e7c-4bca-9659-9bae7235f3d8-kube-api-access-2c82g\" (UniqueName: \"kubernetes.io/projected/2cbcd79a-7e7c-4bca-9659-9bae7235f3d8-kube-api-access-2c82g\") pod \"2cbcd79a-7e7c-4bca-9659-9bae7235f3d8\" (UID: \"2cbcd79a-7e7c-4bca-9659-9bae7235f3d8\") " Oct 30 22:01:12.753477 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:12.753388 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cbcd79a-7e7c-4bca-9659-9bae7235f3d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cbcd79a-7e7c-4bca-9659-9bae7235f3d8-maas-api-xks-ca\") pod \"2cbcd79a-7e7c-4bca-9659-9bae7235f3d8\" (UID: \"2cbcd79a-7e7c-4bca-9659-9bae7235f3d8\") " Oct 30 22:01:12.753518 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:12.753486 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f12246a4-1d55-475b-bcdb-5472da93f641-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f12246a4-1d55-475b-bcdb-5472da93f641-maas-api-openshift-ca\") pod \"f12246a4-1d55-475b-bcdb-5472da93f641\" (UID: \"f12246a4-1d55-475b-bcdb-5472da93f641\") " Oct 30 22:01:12.753560 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:12.753543 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f12246a4-1d55-475b-bcdb-5472da93f641-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f12246a4-1d55-475b-bcdb-5472da93f641-maas-api-xks-ca\") pod \"f12246a4-1d55-475b-bcdb-5472da93f641\" (UID: \"f12246a4-1d55-475b-bcdb-5472da93f641\") " Oct 30 22:01:12.753697 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:12.753670 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cbcd79a-7e7c-4bca-9659-9bae7235f3d8-maas-api-openshift-ca" pod "2cbcd79a-7e7c-4bca-9659-9bae7235f3d8" (UID: "2cbcd79a-7e7c-4bca-9659-9bae7235f3d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:01:12.753877 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:12.753850 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f12246a4-1d55-475b-bcdb-5472da93f641-maas-api-openshift-ca" pod "f12246a4-1d55-475b-bcdb-5472da93f641" (UID: "f12246a4-1d55-475b-bcdb-5472da93f641"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:01:12.754806 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:12.754068 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cbcd79a-7e7c-4bca-9659-9bae7235f3d8-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:12.754806 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:12.754093 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f12246a4-1d55-475b-bcdb-5472da93f641-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:12.755969 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:12.755911 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cbcd79a-7e7c-4bca-9659-9bae7235f3d8-maas-api-xks-ca" pod "2cbcd79a-7e7c-4bca-9659-9bae7235f3d8" (UID: "2cbcd79a-7e7c-4bca-9659-9bae7235f3d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:01:12.756068 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:12.756003 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f12246a4-1d55-475b-bcdb-5472da93f641-maas-api-xks-ca" pod "f12246a4-1d55-475b-bcdb-5472da93f641" (UID: "f12246a4-1d55-475b-bcdb-5472da93f641"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:01:12.756068 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:12.756023 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cbcd79a-7e7c-4bca-9659-9bae7235f3d8-kube-api-access-2c82g" pod "2cbcd79a-7e7c-4bca-9659-9bae7235f3d8" (UID: "2cbcd79a-7e7c-4bca-9659-9bae7235f3d8"). InnerVolumeSpecName "kube-api-access-2c82g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:12.756296 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:12.756270 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f12246a4-1d55-475b-bcdb-5472da93f641-kube-api-access-n8fxg" pod "f12246a4-1d55-475b-bcdb-5472da93f641" (UID: "f12246a4-1d55-475b-bcdb-5472da93f641"). InnerVolumeSpecName "kube-api-access-n8fxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:12.854952 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:12.854921 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8fxg\" (UniqueName: \"kubernetes.io/projected/f12246a4-1d55-475b-bcdb-5472da93f641-kube-api-access-n8fxg\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:12.854952 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:12.854947 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c82g\" (UniqueName: \"kubernetes.io/projected/2cbcd79a-7e7c-4bca-9659-9bae7235f3d8-kube-api-access-2c82g\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:12.854952 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:12.854958 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cbcd79a-7e7c-4bca-9659-9bae7235f3d8-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:12.855168 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:12.854968 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f12246a4-1d55-475b-bcdb-5472da93f641-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:13.438490 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:13.438457 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4e874-284qz" Oct 30 22:01:13.438681 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:13.438494 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4e874-284qz" event={"ID":"2cbcd79a-7e7c-4bca-9659-9bae7235f3d8","Type":"ContainerDied","Data":"76b38c09d8be952b140166b9e3b8272e2ff4c5a0a15996d862abb386a55ae405"} Oct 30 22:01:13.438681 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:13.438555 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76b38c09d8be952b140166b9e3b8272e2ff4c5a0a15996d862abb386a55ae405" Oct 30 22:01:13.439965 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:13.439941 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ca0a55-7ww8s" Oct 30 22:01:13.440070 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:13.439944 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ca0a55-7ww8s" event={"ID":"f12246a4-1d55-475b-bcdb-5472da93f641","Type":"ContainerDied","Data":"07aec596c5f5b01539c9837007959870f5122dd8c695b0b2fdcee489a9ecff37"} Oct 30 22:01:13.440070 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:13.440055 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07aec596c5f5b01539c9837007959870f5122dd8c695b0b2fdcee489a9ecff37" Oct 30 22:01:18.050650 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:18.050616 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76a9fb50-g4z94"] Oct 30 22:01:18.097516 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:18.097487 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76a9fb50-g4z94"] Oct 30 22:01:18.575487 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:18.575455 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51" path="/var/lib/kubelet/pods/317f75b4-a729-4a7b-8bc4-a5d0cbe4ac51/volumes" Oct 30 22:01:19.683737 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:01:19.683696 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:01:21.869310 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:21.869250 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" podUID="e974bae1-bb18-46b0-a4ee-4c01d6455e32" containerName="main" containerID="cri-o://9ba844e55ec42cf1177ef19a3bb2a883454f05d4434cfbaed552f2d7922486cc" gracePeriod=45 Oct 30 22:01:21.887857 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:21.887818 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" podUID="27df5dfe-47a4-4ebb-bd09-1ceef406600a" containerName="main" containerID="cri-o://ca401edad43530267514279db754aebfe262498a484c0261fe962e42b5a9a1a2" gracePeriod=45 Oct 30 22:01:22.163091 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.163061 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" Oct 30 22:01:22.166000 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.165973 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" Oct 30 22:01:22.309949 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.309908 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e974bae1-bb18-46b0-a4ee-4c01d6455e32-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e974bae1-bb18-46b0-a4ee-4c01d6455e32-model-cache\") pod \"e974bae1-bb18-46b0-a4ee-4c01d6455e32\" (UID: \"e974bae1-bb18-46b0-a4ee-4c01d6455e32\") " Oct 30 22:01:22.309949 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.309946 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e974bae1-bb18-46b0-a4ee-4c01d6455e32-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e974bae1-bb18-46b0-a4ee-4c01d6455e32-dshm\") pod \"e974bae1-bb18-46b0-a4ee-4c01d6455e32\" (UID: \"e974bae1-bb18-46b0-a4ee-4c01d6455e32\") " Oct 30 22:01:22.310203 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.310019 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27df5dfe-47a4-4ebb-bd09-1ceef406600a-tls-certs\" (UniqueName: \"kubernetes.io/secret/27df5dfe-47a4-4ebb-bd09-1ceef406600a-tls-certs\") pod \"27df5dfe-47a4-4ebb-bd09-1ceef406600a\" (UID: \"27df5dfe-47a4-4ebb-bd09-1ceef406600a\") " Oct 30 22:01:22.310203 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.310053 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e974bae1-bb18-46b0-a4ee-4c01d6455e32-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e974bae1-bb18-46b0-a4ee-4c01d6455e32-tmp-dir\") pod \"e974bae1-bb18-46b0-a4ee-4c01d6455e32\" (UID: \"e974bae1-bb18-46b0-a4ee-4c01d6455e32\") " Oct 30 22:01:22.310203 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.310104 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27df5dfe-47a4-4ebb-bd09-1ceef406600a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27df5dfe-47a4-4ebb-bd09-1ceef406600a-model-cache\") pod \"27df5dfe-47a4-4ebb-bd09-1ceef406600a\" (UID: \"27df5dfe-47a4-4ebb-bd09-1ceef406600a\") " Oct 30 22:01:22.310203 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.310155 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e974bae1-bb18-46b0-a4ee-4c01d6455e32-kube-api-access-lq6bk\" (UniqueName: \"kubernetes.io/projected/e974bae1-bb18-46b0-a4ee-4c01d6455e32-kube-api-access-lq6bk\") pod \"e974bae1-bb18-46b0-a4ee-4c01d6455e32\" (UID: \"e974bae1-bb18-46b0-a4ee-4c01d6455e32\") " Oct 30 22:01:22.310203 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.310164 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e974bae1-bb18-46b0-a4ee-4c01d6455e32-model-cache" pod "e974bae1-bb18-46b0-a4ee-4c01d6455e32" (UID: "e974bae1-bb18-46b0-a4ee-4c01d6455e32"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:22.310203 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.310181 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27df5dfe-47a4-4ebb-bd09-1ceef406600a-home\" (UniqueName: \"kubernetes.io/empty-dir/27df5dfe-47a4-4ebb-bd09-1ceef406600a-home\") pod \"27df5dfe-47a4-4ebb-bd09-1ceef406600a\" (UID: \"27df5dfe-47a4-4ebb-bd09-1ceef406600a\") " Oct 30 22:01:22.310551 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.310248 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27df5dfe-47a4-4ebb-bd09-1ceef406600a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27df5dfe-47a4-4ebb-bd09-1ceef406600a-tmp-dir\") pod \"27df5dfe-47a4-4ebb-bd09-1ceef406600a\" (UID: \"27df5dfe-47a4-4ebb-bd09-1ceef406600a\") " Oct 30 22:01:22.310551 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.310375 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e974bae1-bb18-46b0-a4ee-4c01d6455e32-home\" (UniqueName: \"kubernetes.io/empty-dir/e974bae1-bb18-46b0-a4ee-4c01d6455e32-home\") pod \"e974bae1-bb18-46b0-a4ee-4c01d6455e32\" (UID: \"e974bae1-bb18-46b0-a4ee-4c01d6455e32\") " Oct 30 22:01:22.310551 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.310412 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27df5dfe-47a4-4ebb-bd09-1ceef406600a-home" pod "27df5dfe-47a4-4ebb-bd09-1ceef406600a" (UID: "27df5dfe-47a4-4ebb-bd09-1ceef406600a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:22.310716 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.310558 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27df5dfe-47a4-4ebb-bd09-1ceef406600a-kube-api-access-5znbk\" (UniqueName: \"kubernetes.io/projected/27df5dfe-47a4-4ebb-bd09-1ceef406600a-kube-api-access-5znbk\") pod \"27df5dfe-47a4-4ebb-bd09-1ceef406600a\" (UID: \"27df5dfe-47a4-4ebb-bd09-1ceef406600a\") " Oct 30 22:01:22.310716 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.310603 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27df5dfe-47a4-4ebb-bd09-1ceef406600a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/27df5dfe-47a4-4ebb-bd09-1ceef406600a-dshm\") pod \"27df5dfe-47a4-4ebb-bd09-1ceef406600a\" (UID: \"27df5dfe-47a4-4ebb-bd09-1ceef406600a\") " Oct 30 22:01:22.310716 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.310621 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e974bae1-bb18-46b0-a4ee-4c01d6455e32-home" pod "e974bae1-bb18-46b0-a4ee-4c01d6455e32" (UID: "e974bae1-bb18-46b0-a4ee-4c01d6455e32"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:22.310716 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.310629 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27df5dfe-47a4-4ebb-bd09-1ceef406600a-tmp-dir" pod "27df5dfe-47a4-4ebb-bd09-1ceef406600a" (UID: "27df5dfe-47a4-4ebb-bd09-1ceef406600a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:22.310716 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.310643 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e974bae1-bb18-46b0-a4ee-4c01d6455e32-tls-certs\" (UniqueName: \"kubernetes.io/secret/e974bae1-bb18-46b0-a4ee-4c01d6455e32-tls-certs\") pod \"e974bae1-bb18-46b0-a4ee-4c01d6455e32\" (UID: \"e974bae1-bb18-46b0-a4ee-4c01d6455e32\") " Oct 30 22:01:22.311212 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.311187 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27df5dfe-47a4-4ebb-bd09-1ceef406600a-home\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:22.311318 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.311217 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27df5dfe-47a4-4ebb-bd09-1ceef406600a-tmp-dir\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:22.311318 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.311232 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e974bae1-bb18-46b0-a4ee-4c01d6455e32-home\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:22.311318 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.311245 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e974bae1-bb18-46b0-a4ee-4c01d6455e32-model-cache\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:22.311704 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.311581 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27df5dfe-47a4-4ebb-bd09-1ceef406600a-model-cache" pod "27df5dfe-47a4-4ebb-bd09-1ceef406600a" (UID: "27df5dfe-47a4-4ebb-bd09-1ceef406600a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:22.311704 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.311669 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e974bae1-bb18-46b0-a4ee-4c01d6455e32-tmp-dir" pod "e974bae1-bb18-46b0-a4ee-4c01d6455e32" (UID: "e974bae1-bb18-46b0-a4ee-4c01d6455e32"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:22.312975 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.312946 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27df5dfe-47a4-4ebb-bd09-1ceef406600a-tls-certs" pod "27df5dfe-47a4-4ebb-bd09-1ceef406600a" (UID: "27df5dfe-47a4-4ebb-bd09-1ceef406600a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:01:22.313060 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.312976 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27df5dfe-47a4-4ebb-bd09-1ceef406600a-dshm" pod "27df5dfe-47a4-4ebb-bd09-1ceef406600a" (UID: "27df5dfe-47a4-4ebb-bd09-1ceef406600a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:22.313060 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.313016 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e974bae1-bb18-46b0-a4ee-4c01d6455e32-kube-api-access-lq6bk" pod "e974bae1-bb18-46b0-a4ee-4c01d6455e32" (UID: "e974bae1-bb18-46b0-a4ee-4c01d6455e32"). InnerVolumeSpecName "kube-api-access-lq6bk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:22.313196 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.313175 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27df5dfe-47a4-4ebb-bd09-1ceef406600a-kube-api-access-5znbk" pod "27df5dfe-47a4-4ebb-bd09-1ceef406600a" (UID: "27df5dfe-47a4-4ebb-bd09-1ceef406600a"). InnerVolumeSpecName "kube-api-access-5znbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:22.314013 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.313992 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e974bae1-bb18-46b0-a4ee-4c01d6455e32-tls-certs" pod "e974bae1-bb18-46b0-a4ee-4c01d6455e32" (UID: "e974bae1-bb18-46b0-a4ee-4c01d6455e32"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:01:22.314268 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.314251 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e974bae1-bb18-46b0-a4ee-4c01d6455e32-dshm" pod "e974bae1-bb18-46b0-a4ee-4c01d6455e32" (UID: "e974bae1-bb18-46b0-a4ee-4c01d6455e32"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:22.469415 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.454634 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27df5dfe-47a4-4ebb-bd09-1ceef406600a-model-cache\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:22.469415 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.454684 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq6bk\" (UniqueName: \"kubernetes.io/projected/e974bae1-bb18-46b0-a4ee-4c01d6455e32-kube-api-access-lq6bk\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:22.469415 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.454705 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5znbk\" (UniqueName: \"kubernetes.io/projected/27df5dfe-47a4-4ebb-bd09-1ceef406600a-kube-api-access-5znbk\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:22.469415 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.454722 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27df5dfe-47a4-4ebb-bd09-1ceef406600a-dshm\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:22.469415 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.454739 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e974bae1-bb18-46b0-a4ee-4c01d6455e32-tls-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:22.469415 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.454753 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e974bae1-bb18-46b0-a4ee-4c01d6455e32-dshm\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:22.469415 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.454768 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27df5dfe-47a4-4ebb-bd09-1ceef406600a-tls-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:22.469415 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.454783 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e974bae1-bb18-46b0-a4ee-4c01d6455e32-tmp-dir\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:22.550671 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.550105 4560 generic.go:356] "Generic (PLEG): container finished" podID="27df5dfe-47a4-4ebb-bd09-1ceef406600a" containerID="ca401edad43530267514279db754aebfe262498a484c0261fe962e42b5a9a1a2" exitCode=0 Oct 30 22:01:22.550671 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.550179 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" Oct 30 22:01:22.550671 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.550191 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" event={"ID":"27df5dfe-47a4-4ebb-bd09-1ceef406600a","Type":"ContainerDied","Data":"ca401edad43530267514279db754aebfe262498a484c0261fe962e42b5a9a1a2"} Oct 30 22:01:22.550671 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.550240 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm" event={"ID":"27df5dfe-47a4-4ebb-bd09-1ceef406600a","Type":"ContainerDied","Data":"a08068f3131a903b082fcbb9720579569ecc4feefa6aa74c0e1d70dac4d77c2e"} Oct 30 22:01:22.550671 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.550263 4560 scope.go:122] "RemoveContainer" containerID="ca401edad43530267514279db754aebfe262498a484c0261fe962e42b5a9a1a2" Oct 30 22:01:22.552845 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.552497 4560 generic.go:356] "Generic (PLEG): container finished" podID="e974bae1-bb18-46b0-a4ee-4c01d6455e32" containerID="9ba844e55ec42cf1177ef19a3bb2a883454f05d4434cfbaed552f2d7922486cc" exitCode=0 Oct 30 22:01:22.552845 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.552599 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" event={"ID":"e974bae1-bb18-46b0-a4ee-4c01d6455e32","Type":"ContainerDied","Data":"9ba844e55ec42cf1177ef19a3bb2a883454f05d4434cfbaed552f2d7922486cc"} Oct 30 22:01:22.552845 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.552628 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" event={"ID":"e974bae1-bb18-46b0-a4ee-4c01d6455e32","Type":"ContainerDied","Data":"81bd16177b46aa77a2a102dffc6948e89db134f3307ef8df47a23dc713536613"} Oct 30 22:01:22.552845 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.552686 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5" Oct 30 22:01:22.577895 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.577876 4560 scope.go:122] "RemoveContainer" containerID="ca401edad43530267514279db754aebfe262498a484c0261fe962e42b5a9a1a2" Oct 30 22:01:22.578190 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:01:22.578164 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca401edad43530267514279db754aebfe262498a484c0261fe962e42b5a9a1a2\": container with ID starting with ca401edad43530267514279db754aebfe262498a484c0261fe962e42b5a9a1a2 not found: ID does not exist" containerID="ca401edad43530267514279db754aebfe262498a484c0261fe962e42b5a9a1a2" Oct 30 22:01:22.578289 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.578200 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca401edad43530267514279db754aebfe262498a484c0261fe962e42b5a9a1a2"} err="failed to get container status \"ca401edad43530267514279db754aebfe262498a484c0261fe962e42b5a9a1a2\": rpc error: code = NotFound desc = could not find container \"ca401edad43530267514279db754aebfe262498a484c0261fe962e42b5a9a1a2\": container with ID starting with ca401edad43530267514279db754aebfe262498a484c0261fe962e42b5a9a1a2 not found: ID does not exist" Oct 30 22:01:22.578289 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.578224 4560 scope.go:122] "RemoveContainer" containerID="9ba844e55ec42cf1177ef19a3bb2a883454f05d4434cfbaed552f2d7922486cc" Oct 30 22:01:22.587417 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.587396 4560 scope.go:122] "RemoveContainer" containerID="9ba844e55ec42cf1177ef19a3bb2a883454f05d4434cfbaed552f2d7922486cc" Oct 30 22:01:22.587671 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:01:22.587654 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ba844e55ec42cf1177ef19a3bb2a883454f05d4434cfbaed552f2d7922486cc\": container with ID starting with 9ba844e55ec42cf1177ef19a3bb2a883454f05d4434cfbaed552f2d7922486cc not found: ID does not exist" containerID="9ba844e55ec42cf1177ef19a3bb2a883454f05d4434cfbaed552f2d7922486cc" Oct 30 22:01:22.587739 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.587678 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ba844e55ec42cf1177ef19a3bb2a883454f05d4434cfbaed552f2d7922486cc"} err="failed to get container status \"9ba844e55ec42cf1177ef19a3bb2a883454f05d4434cfbaed552f2d7922486cc\": rpc error: code = NotFound desc = could not find container \"9ba844e55ec42cf1177ef19a3bb2a883454f05d4434cfbaed552f2d7922486cc\": container with ID starting with 9ba844e55ec42cf1177ef19a3bb2a883454f05d4434cfbaed552f2d7922486cc not found: ID does not exist" Oct 30 22:01:22.720611 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.720523 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5"] Oct 30 22:01:22.727635 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.727606 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-19dd4f/distinct-w1-19dd4f-kserve-798f8b8b56-pxct5"] Oct 30 22:01:22.841033 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.841001 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm"] Oct 30 22:01:22.846811 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:22.846784 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-19dd4f/simulator-w1-19dd4f-kserve-674558b488-b4pmm"] Oct 30 22:01:24.583310 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:24.583273 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27df5dfe-47a4-4ebb-bd09-1ceef406600a" path="/var/lib/kubelet/pods/27df5dfe-47a4-4ebb-bd09-1ceef406600a/volumes" Oct 30 22:01:24.583927 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:24.583910 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e974bae1-bb18-46b0-a4ee-4c01d6455e32" path="/var/lib/kubelet/pods/e974bae1-bb18-46b0-a4ee-4c01d6455e32/volumes" Oct 30 22:01:26.486866 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:26.486824 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-kfkwt"] Oct 30 22:01:26.533451 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:26.533421 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bfdf30b9-kfkwt"] Oct 30 22:01:26.575215 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:26.575180 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d5d0829-37d1-4139-8c40-6c7cc3ba92b1" path="/var/lib/kubelet/pods/6d5d0829-37d1-4139-8c40-6c7cc3ba92b1/volumes" Oct 30 22:01:31.909411 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:31.909378 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-bfdf30b9/e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc"] Oct 30 22:01:32.048448 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:32.048413 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23808097-4s8n8"] Oct 30 22:01:32.070493 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:32.070468 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23808097-4s8n8"] Oct 30 22:01:32.575243 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:32.575211 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32d2830c-c92e-488a-a542-3eb9145cceab" path="/var/lib/kubelet/pods/32d2830c-c92e-488a-a542-3eb9145cceab/volumes" Oct 30 22:01:40.873709 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:40.873672 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:01:45.252555 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.252517 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfe1cd9b-dxgzt"] Oct 30 22:01:45.253070 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.253055 4560 cpu_manager.go:414] "containerMap: removing container" podUID="e974bae1-bb18-46b0-a4ee-4c01d6455e32" containerName="main" Oct 30 22:01:45.253113 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.253071 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e974bae1-bb18-46b0-a4ee-4c01d6455e32" containerName="main" Oct 30 22:01:45.253113 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.253088 4560 cpu_manager.go:414] "containerMap: removing container" podUID="2cbcd79a-7e7c-4bca-9659-9bae7235f3d8" containerName="revoke-keys" Oct 30 22:01:45.253113 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.253093 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cbcd79a-7e7c-4bca-9659-9bae7235f3d8" containerName="revoke-keys" Oct 30 22:01:45.253113 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.253105 4560 cpu_manager.go:414] "containerMap: removing container" podUID="27df5dfe-47a4-4ebb-bd09-1ceef406600a" containerName="main" Oct 30 22:01:45.253113 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.253110 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27df5dfe-47a4-4ebb-bd09-1ceef406600a" containerName="main" Oct 30 22:01:45.253272 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.253117 4560 cpu_manager.go:414] "containerMap: removing container" podUID="f12246a4-1d55-475b-bcdb-5472da93f641" containerName="revoke-keys" Oct 30 22:01:45.253272 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.253122 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f12246a4-1d55-475b-bcdb-5472da93f641" containerName="revoke-keys" Oct 30 22:01:45.253272 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.253194 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="f12246a4-1d55-475b-bcdb-5472da93f641" containerName="revoke-keys" Oct 30 22:01:45.253272 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.253206 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="e974bae1-bb18-46b0-a4ee-4c01d6455e32" containerName="main" Oct 30 22:01:45.253272 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.253212 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cbcd79a-7e7c-4bca-9659-9bae7235f3d8" containerName="revoke-keys" Oct 30 22:01:45.253272 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.253219 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="27df5dfe-47a4-4ebb-bd09-1ceef406600a" containerName="main" Oct 30 22:01:45.256396 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.256376 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe1cd9b-dxgzt" Oct 30 22:01:45.303382 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.303351 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-816ca17c-l2249"] Oct 30 22:01:45.307059 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.307039 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-816ca17c-l2249" Oct 30 22:01:45.356071 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.356028 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:01:45.356071 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.356029 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:01:45.356741 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.356714 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b0e07f4-337a-4961-aa14-36f1c8989433-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfe1cd9b-dxgzt\" (UID: \"2b0e07f4-337a-4961-aa14-36f1c8989433\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe1cd9b-dxgzt" Oct 30 22:01:45.356865 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.356763 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzscv\" (UniqueName: \"kubernetes.io/projected/2b0e07f4-337a-4961-aa14-36f1c8989433-kube-api-access-gzscv\") pod \"maas-api-revoke-sub-dfe1cd9b-dxgzt\" (UID: \"2b0e07f4-337a-4961-aa14-36f1c8989433\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe1cd9b-dxgzt" Oct 30 22:01:45.357017 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.356997 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b0e07f4-337a-4961-aa14-36f1c8989433-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfe1cd9b-dxgzt\" (UID: \"2b0e07f4-337a-4961-aa14-36f1c8989433\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe1cd9b-dxgzt" Oct 30 22:01:45.357488 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.357468 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfe1cd9b-dxgzt"] Oct 30 22:01:45.365800 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.365780 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:01:45.365800 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.365798 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:01:45.375896 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.375870 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-816ca17c-l2249"] Oct 30 22:01:45.457845 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.457808 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b0e07f4-337a-4961-aa14-36f1c8989433-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfe1cd9b-dxgzt\" (UID: \"2b0e07f4-337a-4961-aa14-36f1c8989433\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe1cd9b-dxgzt" Oct 30 22:01:45.458065 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.457867 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzscv\" (UniqueName: \"kubernetes.io/projected/2b0e07f4-337a-4961-aa14-36f1c8989433-kube-api-access-gzscv\") pod \"maas-api-revoke-sub-dfe1cd9b-dxgzt\" (UID: \"2b0e07f4-337a-4961-aa14-36f1c8989433\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe1cd9b-dxgzt" Oct 30 22:01:45.458065 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.457964 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11aa0088-fac9-4ff4-a925-e241e613137f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-816ca17c-l2249\" (UID: \"11aa0088-fac9-4ff4-a925-e241e613137f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-816ca17c-l2249" Oct 30 22:01:45.458065 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.458020 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mrzv\" (UniqueName: \"kubernetes.io/projected/11aa0088-fac9-4ff4-a925-e241e613137f-kube-api-access-7mrzv\") pod \"maas-api-revoke-sub-816ca17c-l2249\" (UID: \"11aa0088-fac9-4ff4-a925-e241e613137f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-816ca17c-l2249" Oct 30 22:01:45.458273 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.458116 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11aa0088-fac9-4ff4-a925-e241e613137f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-816ca17c-l2249\" (UID: \"11aa0088-fac9-4ff4-a925-e241e613137f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-816ca17c-l2249" Oct 30 22:01:45.458273 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.458179 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b0e07f4-337a-4961-aa14-36f1c8989433-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfe1cd9b-dxgzt\" (UID: \"2b0e07f4-337a-4961-aa14-36f1c8989433\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe1cd9b-dxgzt" Oct 30 22:01:45.458807 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.458783 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b0e07f4-337a-4961-aa14-36f1c8989433-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfe1cd9b-dxgzt\" (UID: \"2b0e07f4-337a-4961-aa14-36f1c8989433\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe1cd9b-dxgzt" Oct 30 22:01:45.460727 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.460704 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b0e07f4-337a-4961-aa14-36f1c8989433-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfe1cd9b-dxgzt\" (UID: \"2b0e07f4-337a-4961-aa14-36f1c8989433\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe1cd9b-dxgzt" Oct 30 22:01:45.518685 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.518624 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzscv\" (UniqueName: \"kubernetes.io/projected/2b0e07f4-337a-4961-aa14-36f1c8989433-kube-api-access-gzscv\") pod \"maas-api-revoke-sub-dfe1cd9b-dxgzt\" (UID: \"2b0e07f4-337a-4961-aa14-36f1c8989433\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe1cd9b-dxgzt" Oct 30 22:01:45.560630 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.560260 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11aa0088-fac9-4ff4-a925-e241e613137f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-816ca17c-l2249\" (UID: \"11aa0088-fac9-4ff4-a925-e241e613137f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-816ca17c-l2249" Oct 30 22:01:45.560630 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.560356 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mrzv\" (UniqueName: \"kubernetes.io/projected/11aa0088-fac9-4ff4-a925-e241e613137f-kube-api-access-7mrzv\") pod \"maas-api-revoke-sub-816ca17c-l2249\" (UID: \"11aa0088-fac9-4ff4-a925-e241e613137f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-816ca17c-l2249" Oct 30 22:01:45.560630 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.560447 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11aa0088-fac9-4ff4-a925-e241e613137f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-816ca17c-l2249\" (UID: \"11aa0088-fac9-4ff4-a925-e241e613137f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-816ca17c-l2249" Oct 30 22:01:45.561271 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.561197 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11aa0088-fac9-4ff4-a925-e241e613137f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-816ca17c-l2249\" (UID: \"11aa0088-fac9-4ff4-a925-e241e613137f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-816ca17c-l2249" Oct 30 22:01:45.563822 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.563798 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11aa0088-fac9-4ff4-a925-e241e613137f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-816ca17c-l2249\" (UID: \"11aa0088-fac9-4ff4-a925-e241e613137f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-816ca17c-l2249" Oct 30 22:01:45.571063 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.570814 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe1cd9b-dxgzt" Oct 30 22:01:45.627233 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.627200 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mrzv\" (UniqueName: \"kubernetes.io/projected/11aa0088-fac9-4ff4-a925-e241e613137f-kube-api-access-7mrzv\") pod \"maas-api-revoke-sub-816ca17c-l2249\" (UID: \"11aa0088-fac9-4ff4-a925-e241e613137f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-816ca17c-l2249" Oct 30 22:01:45.933061 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:45.931661 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-816ca17c-l2249" Oct 30 22:01:46.158437 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:46.158403 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfe1cd9b-dxgzt"] Oct 30 22:01:46.508376 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:46.508329 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-816ca17c-l2249"] Oct 30 22:01:46.514538 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:01:46.514512 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11aa0088_fac9_4ff4_a925_e241e613137f.slice/crio-56118db5d6d1421826b3c23a81315c35e8b355608caab65458faaf9e3d02b3c6 WatchSource:0}: Error finding container 56118db5d6d1421826b3c23a81315c35e8b355608caab65458faaf9e3d02b3c6: Status 404 returned error can't find the container with id 56118db5d6d1421826b3c23a81315c35e8b355608caab65458faaf9e3d02b3c6 Oct 30 22:01:46.653434 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:46.652155 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd387d1e-lpzbb"] Oct 30 22:01:46.656348 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:46.656316 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd387d1e-lpzbb" Oct 30 22:01:46.693023 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:46.692998 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c34bf965-stnn7"] Oct 30 22:01:46.696613 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:46.696596 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c34bf965-stnn7" Oct 30 22:01:46.703720 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:46.703694 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd387d1e-lpzbb"] Oct 30 22:01:46.725284 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:46.725257 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq9qm\" (UniqueName: \"kubernetes.io/projected/282c407f-8be5-4c81-9d4c-e67f7ba44cc9-kube-api-access-zq9qm\") pod \"maas-api-revoke-sub-dd387d1e-lpzbb\" (UID: \"282c407f-8be5-4c81-9d4c-e67f7ba44cc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd387d1e-lpzbb" Oct 30 22:01:46.725406 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:46.725304 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ee952bc-3471-4387-8b50-b242f8dc9b44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c34bf965-stnn7\" (UID: \"5ee952bc-3471-4387-8b50-b242f8dc9b44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c34bf965-stnn7" Oct 30 22:01:46.725456 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:46.725419 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h6ds\" (UniqueName: \"kubernetes.io/projected/5ee952bc-3471-4387-8b50-b242f8dc9b44-kube-api-access-4h6ds\") pod \"maas-api-revoke-sub-c34bf965-stnn7\" (UID: \"5ee952bc-3471-4387-8b50-b242f8dc9b44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c34bf965-stnn7" Oct 30 22:01:46.725492 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:46.725484 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ee952bc-3471-4387-8b50-b242f8dc9b44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c34bf965-stnn7\" (UID: \"5ee952bc-3471-4387-8b50-b242f8dc9b44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c34bf965-stnn7" Oct 30 22:01:46.725556 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:46.725542 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/282c407f-8be5-4c81-9d4c-e67f7ba44cc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd387d1e-lpzbb\" (UID: \"282c407f-8be5-4c81-9d4c-e67f7ba44cc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd387d1e-lpzbb" Oct 30 22:01:46.725624 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:46.725612 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/282c407f-8be5-4c81-9d4c-e67f7ba44cc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd387d1e-lpzbb\" (UID: \"282c407f-8be5-4c81-9d4c-e67f7ba44cc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd387d1e-lpzbb" Oct 30 22:01:46.759934 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:46.759893 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c34bf965-stnn7"] Oct 30 22:01:46.819805 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:46.819765 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe1cd9b-dxgzt" event={"ID":"2b0e07f4-337a-4961-aa14-36f1c8989433","Type":"ContainerStarted","Data":"fc3933595d4530175cfb00191bdb08b35869c477a28af454d450895e2ccdbc85"} Oct 30 22:01:46.819991 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:46.819822 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe1cd9b-dxgzt" event={"ID":"2b0e07f4-337a-4961-aa14-36f1c8989433","Type":"ContainerStarted","Data":"0fffd5cd4bcd81c9ab8c10282699e1623e7b58858ce7efebdde883e6332e1b1c"} Oct 30 22:01:46.821086 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:46.821064 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-816ca17c-l2249" event={"ID":"11aa0088-fac9-4ff4-a925-e241e613137f","Type":"ContainerStarted","Data":"56118db5d6d1421826b3c23a81315c35e8b355608caab65458faaf9e3d02b3c6"} Oct 30 22:01:46.850255 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:46.843249 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/282c407f-8be5-4c81-9d4c-e67f7ba44cc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd387d1e-lpzbb\" (UID: \"282c407f-8be5-4c81-9d4c-e67f7ba44cc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd387d1e-lpzbb" Oct 30 22:01:46.850255 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:46.843411 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq9qm\" (UniqueName: \"kubernetes.io/projected/282c407f-8be5-4c81-9d4c-e67f7ba44cc9-kube-api-access-zq9qm\") pod \"maas-api-revoke-sub-dd387d1e-lpzbb\" (UID: \"282c407f-8be5-4c81-9d4c-e67f7ba44cc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd387d1e-lpzbb" Oct 30 22:01:46.850255 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:46.843463 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ee952bc-3471-4387-8b50-b242f8dc9b44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c34bf965-stnn7\" (UID: \"5ee952bc-3471-4387-8b50-b242f8dc9b44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c34bf965-stnn7" Oct 30 22:01:46.850255 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:46.843515 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h6ds\" (UniqueName: \"kubernetes.io/projected/5ee952bc-3471-4387-8b50-b242f8dc9b44-kube-api-access-4h6ds\") pod \"maas-api-revoke-sub-c34bf965-stnn7\" (UID: \"5ee952bc-3471-4387-8b50-b242f8dc9b44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c34bf965-stnn7" Oct 30 22:01:46.850255 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:46.843592 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ee952bc-3471-4387-8b50-b242f8dc9b44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c34bf965-stnn7\" (UID: \"5ee952bc-3471-4387-8b50-b242f8dc9b44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c34bf965-stnn7" Oct 30 22:01:46.850255 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:46.843662 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/282c407f-8be5-4c81-9d4c-e67f7ba44cc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd387d1e-lpzbb\" (UID: \"282c407f-8be5-4c81-9d4c-e67f7ba44cc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd387d1e-lpzbb" Oct 30 22:01:46.850720 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:46.850256 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/282c407f-8be5-4c81-9d4c-e67f7ba44cc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd387d1e-lpzbb\" (UID: \"282c407f-8be5-4c81-9d4c-e67f7ba44cc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd387d1e-lpzbb" Oct 30 22:01:46.852360 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:46.852033 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ee952bc-3471-4387-8b50-b242f8dc9b44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c34bf965-stnn7\" (UID: \"5ee952bc-3471-4387-8b50-b242f8dc9b44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c34bf965-stnn7" Oct 30 22:01:46.855214 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:46.855175 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/282c407f-8be5-4c81-9d4c-e67f7ba44cc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd387d1e-lpzbb\" (UID: \"282c407f-8be5-4c81-9d4c-e67f7ba44cc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd387d1e-lpzbb" Oct 30 22:01:46.855780 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:46.855753 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ee952bc-3471-4387-8b50-b242f8dc9b44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c34bf965-stnn7\" (UID: \"5ee952bc-3471-4387-8b50-b242f8dc9b44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c34bf965-stnn7" Oct 30 22:01:46.909978 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:46.909921 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h6ds\" (UniqueName: \"kubernetes.io/projected/5ee952bc-3471-4387-8b50-b242f8dc9b44-kube-api-access-4h6ds\") pod \"maas-api-revoke-sub-c34bf965-stnn7\" (UID: \"5ee952bc-3471-4387-8b50-b242f8dc9b44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c34bf965-stnn7" Oct 30 22:01:46.910163 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:46.910099 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq9qm\" (UniqueName: \"kubernetes.io/projected/282c407f-8be5-4c81-9d4c-e67f7ba44cc9-kube-api-access-zq9qm\") pod \"maas-api-revoke-sub-dd387d1e-lpzbb\" (UID: \"282c407f-8be5-4c81-9d4c-e67f7ba44cc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd387d1e-lpzbb" Oct 30 22:01:46.949371 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:46.947510 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-bfdf30b9/e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc" podUID="90966441-80bf-4bc5-8e68-4dea00d9f779" containerName="main" containerID="cri-o://b1c4594f3a30dcbfb66af51cfbfbc5300f208b15e8d641ca2999e5a644223b81" gracePeriod=45 Oct 30 22:01:46.968888 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:46.968866 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd387d1e-lpzbb" Oct 30 22:01:47.019161 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.018839 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c34bf965-stnn7" Oct 30 22:01:47.059490 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.049967 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe1cd9b-dxgzt" podStartSLOduration=2.049942363 podStartE2EDuration="2.049942363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:47.022427025 +0000 UTC m=+2882.577126836" watchObservedRunningTime="2026-10-30 22:01:47.049942363 +0000 UTC m=+2882.604642140" Oct 30 22:01:47.161597 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.160265 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-bfdf30b9/e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc" podUID="90966441-80bf-4bc5-8e68-4dea00d9f779" containerName="main" probeResult="failure" output="Get \"https://10.133.0.134:8000/ready\": dial tcp 10.133.0.134:8000: connect: connection refused" Oct 30 22:01:47.323508 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.323482 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bfdf30b9/e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc" Oct 30 22:01:47.387659 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.386282 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd387d1e-lpzbb"] Oct 30 22:01:47.388516 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:01:47.388493 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod282c407f_8be5_4c81_9d4c_e67f7ba44cc9.slice/crio-72659e7c79cff47ae5c1fbb6753fed38c0764ab9604a06cebe5e5218380f18ae WatchSource:0}: Error finding container 72659e7c79cff47ae5c1fbb6753fed38c0764ab9604a06cebe5e5218380f18ae: Status 404 returned error can't find the container with id 72659e7c79cff47ae5c1fbb6753fed38c0764ab9604a06cebe5e5218380f18ae Oct 30 22:01:47.472907 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.472852 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90966441-80bf-4bc5-8e68-4dea00d9f779-tls-certs\" (UniqueName: \"kubernetes.io/secret/90966441-80bf-4bc5-8e68-4dea00d9f779-tls-certs\") pod \"90966441-80bf-4bc5-8e68-4dea00d9f779\" (UID: \"90966441-80bf-4bc5-8e68-4dea00d9f779\") " Oct 30 22:01:47.473083 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.472949 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90966441-80bf-4bc5-8e68-4dea00d9f779-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90966441-80bf-4bc5-8e68-4dea00d9f779-tmp-dir\") pod \"90966441-80bf-4bc5-8e68-4dea00d9f779\" (UID: \"90966441-80bf-4bc5-8e68-4dea00d9f779\") " Oct 30 22:01:47.473083 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.473010 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90966441-80bf-4bc5-8e68-4dea00d9f779-kube-api-access-4fdx6\" (UniqueName: \"kubernetes.io/projected/90966441-80bf-4bc5-8e68-4dea00d9f779-kube-api-access-4fdx6\") pod \"90966441-80bf-4bc5-8e68-4dea00d9f779\" (UID: \"90966441-80bf-4bc5-8e68-4dea00d9f779\") " Oct 30 22:01:47.473083 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.473062 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90966441-80bf-4bc5-8e68-4dea00d9f779-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90966441-80bf-4bc5-8e68-4dea00d9f779-model-cache\") pod \"90966441-80bf-4bc5-8e68-4dea00d9f779\" (UID: \"90966441-80bf-4bc5-8e68-4dea00d9f779\") " Oct 30 22:01:47.473255 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.473145 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90966441-80bf-4bc5-8e68-4dea00d9f779-home\" (UniqueName: \"kubernetes.io/empty-dir/90966441-80bf-4bc5-8e68-4dea00d9f779-home\") pod \"90966441-80bf-4bc5-8e68-4dea00d9f779\" (UID: \"90966441-80bf-4bc5-8e68-4dea00d9f779\") " Oct 30 22:01:47.473255 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.473188 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90966441-80bf-4bc5-8e68-4dea00d9f779-dshm\" (UniqueName: \"kubernetes.io/empty-dir/90966441-80bf-4bc5-8e68-4dea00d9f779-dshm\") pod \"90966441-80bf-4bc5-8e68-4dea00d9f779\" (UID: \"90966441-80bf-4bc5-8e68-4dea00d9f779\") " Oct 30 22:01:47.473387 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.473255 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90966441-80bf-4bc5-8e68-4dea00d9f779-tmp-dir" pod "90966441-80bf-4bc5-8e68-4dea00d9f779" (UID: "90966441-80bf-4bc5-8e68-4dea00d9f779"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:47.473528 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.473506 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90966441-80bf-4bc5-8e68-4dea00d9f779-model-cache" pod "90966441-80bf-4bc5-8e68-4dea00d9f779" (UID: "90966441-80bf-4bc5-8e68-4dea00d9f779"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:47.473695 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.473671 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90966441-80bf-4bc5-8e68-4dea00d9f779-home" pod "90966441-80bf-4bc5-8e68-4dea00d9f779" (UID: "90966441-80bf-4bc5-8e68-4dea00d9f779"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:47.473846 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.473824 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90966441-80bf-4bc5-8e68-4dea00d9f779-tmp-dir\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:47.473846 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.473844 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90966441-80bf-4bc5-8e68-4dea00d9f779-model-cache\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:47.474039 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.473854 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90966441-80bf-4bc5-8e68-4dea00d9f779-home\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:47.476203 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.476181 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90966441-80bf-4bc5-8e68-4dea00d9f779-dshm" pod "90966441-80bf-4bc5-8e68-4dea00d9f779" (UID: "90966441-80bf-4bc5-8e68-4dea00d9f779"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:01:47.476434 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.476414 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90966441-80bf-4bc5-8e68-4dea00d9f779-tls-certs" pod "90966441-80bf-4bc5-8e68-4dea00d9f779" (UID: "90966441-80bf-4bc5-8e68-4dea00d9f779"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:01:47.477432 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.477358 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90966441-80bf-4bc5-8e68-4dea00d9f779-kube-api-access-4fdx6" pod "90966441-80bf-4bc5-8e68-4dea00d9f779" (UID: "90966441-80bf-4bc5-8e68-4dea00d9f779"). InnerVolumeSpecName "kube-api-access-4fdx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:47.497760 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.497730 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c34bf965-stnn7"] Oct 30 22:01:47.504492 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:01:47.504430 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ee952bc_3471_4387_8b50_b242f8dc9b44.slice/crio-b5d5cf1e3273cac89cfae295fc83116b55f8268c3cef21b54251f7c139367e31 WatchSource:0}: Error finding container b5d5cf1e3273cac89cfae295fc83116b55f8268c3cef21b54251f7c139367e31: Status 404 returned error can't find the container with id b5d5cf1e3273cac89cfae295fc83116b55f8268c3cef21b54251f7c139367e31 Oct 30 22:01:47.579284 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.574714 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fdx6\" (UniqueName: \"kubernetes.io/projected/90966441-80bf-4bc5-8e68-4dea00d9f779-kube-api-access-4fdx6\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:47.579284 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.574748 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90966441-80bf-4bc5-8e68-4dea00d9f779-dshm\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:47.579284 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.574763 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90966441-80bf-4bc5-8e68-4dea00d9f779-tls-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:47.580195 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.580169 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4e874-284qz"] Oct 30 22:01:47.583366 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.583327 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4e874-284qz"] Oct 30 22:01:47.840864 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.840630 4560 generic.go:356] "Generic (PLEG): container finished" podID="11aa0088-fac9-4ff4-a925-e241e613137f" containerID="f1509cae3d313d33ceb8a50c0c8dcda43c31a3f20c4302854ab7c6372dc1db1d" exitCode=0 Oct 30 22:01:47.840864 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.840789 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-816ca17c-l2249" event={"ID":"11aa0088-fac9-4ff4-a925-e241e613137f","Type":"ContainerDied","Data":"f1509cae3d313d33ceb8a50c0c8dcda43c31a3f20c4302854ab7c6372dc1db1d"} Oct 30 22:01:47.843470 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.843434 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c34bf965-stnn7" event={"ID":"5ee952bc-3471-4387-8b50-b242f8dc9b44","Type":"ContainerStarted","Data":"b5d5cf1e3273cac89cfae295fc83116b55f8268c3cef21b54251f7c139367e31"} Oct 30 22:01:47.845373 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.845331 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd387d1e-lpzbb" event={"ID":"282c407f-8be5-4c81-9d4c-e67f7ba44cc9","Type":"ContainerStarted","Data":"5bc41345b066f183450dfafd3942c3c4d8cf8845bfa7d7c71218ce71b0056d34"} Oct 30 22:01:47.845508 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.845395 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd387d1e-lpzbb" event={"ID":"282c407f-8be5-4c81-9d4c-e67f7ba44cc9","Type":"ContainerStarted","Data":"72659e7c79cff47ae5c1fbb6753fed38c0764ab9604a06cebe5e5218380f18ae"} Oct 30 22:01:47.846705 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.846683 4560 generic.go:356] "Generic (PLEG): container finished" podID="90966441-80bf-4bc5-8e68-4dea00d9f779" containerID="b1c4594f3a30dcbfb66af51cfbfbc5300f208b15e8d641ca2999e5a644223b81" exitCode=0 Oct 30 22:01:47.846803 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.846731 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bfdf30b9/e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc" Oct 30 22:01:47.846803 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.846754 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bfdf30b9/e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc" event={"ID":"90966441-80bf-4bc5-8e68-4dea00d9f779","Type":"ContainerDied","Data":"b1c4594f3a30dcbfb66af51cfbfbc5300f208b15e8d641ca2999e5a644223b81"} Oct 30 22:01:47.846914 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.846805 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bfdf30b9/e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc" event={"ID":"90966441-80bf-4bc5-8e68-4dea00d9f779","Type":"ContainerDied","Data":"6f742e70a474155f8f9cbdd2a4c8d9393527ea10d4c6de117201bf84882d9b04"} Oct 30 22:01:47.846914 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.846830 4560 scope.go:122] "RemoveContainer" containerID="b1c4594f3a30dcbfb66af51cfbfbc5300f208b15e8d641ca2999e5a644223b81" Oct 30 22:01:47.848498 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.848474 4560 generic.go:356] "Generic (PLEG): container finished" podID="2b0e07f4-337a-4961-aa14-36f1c8989433" containerID="fc3933595d4530175cfb00191bdb08b35869c477a28af454d450895e2ccdbc85" exitCode=0 Oct 30 22:01:47.848612 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.848513 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe1cd9b-dxgzt" event={"ID":"2b0e07f4-337a-4961-aa14-36f1c8989433","Type":"ContainerDied","Data":"fc3933595d4530175cfb00191bdb08b35869c477a28af454d450895e2ccdbc85"} Oct 30 22:01:47.859844 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.859827 4560 scope.go:122] "RemoveContainer" containerID="b1c4594f3a30dcbfb66af51cfbfbc5300f208b15e8d641ca2999e5a644223b81" Oct 30 22:01:47.860155 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:01:47.860135 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1c4594f3a30dcbfb66af51cfbfbc5300f208b15e8d641ca2999e5a644223b81\": container with ID starting with b1c4594f3a30dcbfb66af51cfbfbc5300f208b15e8d641ca2999e5a644223b81 not found: ID does not exist" containerID="b1c4594f3a30dcbfb66af51cfbfbc5300f208b15e8d641ca2999e5a644223b81" Oct 30 22:01:47.860210 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:47.860165 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1c4594f3a30dcbfb66af51cfbfbc5300f208b15e8d641ca2999e5a644223b81"} err="failed to get container status \"b1c4594f3a30dcbfb66af51cfbfbc5300f208b15e8d641ca2999e5a644223b81\": rpc error: code = NotFound desc = could not find container \"b1c4594f3a30dcbfb66af51cfbfbc5300f208b15e8d641ca2999e5a644223b81\": container with ID starting with b1c4594f3a30dcbfb66af51cfbfbc5300f208b15e8d641ca2999e5a644223b81 not found: ID does not exist" Oct 30 22:01:48.074708 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:48.074652 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd387d1e-lpzbb" podStartSLOduration=2.07463717 podStartE2EDuration="2.07463717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:48.072372318 +0000 UTC m=+2883.627072088" watchObservedRunningTime="2026-10-30 22:01:48.07463717 +0000 UTC m=+2883.629336946" Oct 30 22:01:48.288348 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:48.288309 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-bfdf30b9/e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc"] Oct 30 22:01:48.294560 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:48.294532 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-bfdf30b9/e2e-lifecycle-model-bfdf30b9-kserve-854b8c849c-csfjc"] Oct 30 22:01:48.578153 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:48.578083 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cbcd79a-7e7c-4bca-9659-9bae7235f3d8" path="/var/lib/kubelet/pods/2cbcd79a-7e7c-4bca-9659-9bae7235f3d8/volumes" Oct 30 22:01:48.578571 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:48.578555 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90966441-80bf-4bc5-8e68-4dea00d9f779" path="/var/lib/kubelet/pods/90966441-80bf-4bc5-8e68-4dea00d9f779/volumes" Oct 30 22:01:48.857391 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:48.857290 4560 generic.go:356] "Generic (PLEG): container finished" podID="5ee952bc-3471-4387-8b50-b242f8dc9b44" containerID="5be1a42105ef388be038cc01d603f0b7532978e5def79a215a2fdcc45776e0fe" exitCode=0 Oct 30 22:01:48.857825 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:48.857380 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c34bf965-stnn7" event={"ID":"5ee952bc-3471-4387-8b50-b242f8dc9b44","Type":"ContainerDied","Data":"5be1a42105ef388be038cc01d603f0b7532978e5def79a215a2fdcc45776e0fe"} Oct 30 22:01:48.859078 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:48.859049 4560 generic.go:356] "Generic (PLEG): container finished" podID="282c407f-8be5-4c81-9d4c-e67f7ba44cc9" containerID="5bc41345b066f183450dfafd3942c3c4d8cf8845bfa7d7c71218ce71b0056d34" exitCode=0 Oct 30 22:01:48.859208 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:48.859119 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd387d1e-lpzbb" event={"ID":"282c407f-8be5-4c81-9d4c-e67f7ba44cc9","Type":"ContainerDied","Data":"5bc41345b066f183450dfafd3942c3c4d8cf8845bfa7d7c71218ce71b0056d34"} Oct 30 22:01:49.039419 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:49.039389 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe1cd9b-dxgzt" Oct 30 22:01:49.078644 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:49.078621 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-816ca17c-l2249" Oct 30 22:01:49.099380 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:49.099334 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11aa0088-fac9-4ff4-a925-e241e613137f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11aa0088-fac9-4ff4-a925-e241e613137f-maas-api-xks-ca\") pod \"11aa0088-fac9-4ff4-a925-e241e613137f\" (UID: \"11aa0088-fac9-4ff4-a925-e241e613137f\") " Oct 30 22:01:49.099553 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:49.099396 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b0e07f4-337a-4961-aa14-36f1c8989433-kube-api-access-gzscv\" (UniqueName: \"kubernetes.io/projected/2b0e07f4-337a-4961-aa14-36f1c8989433-kube-api-access-gzscv\") pod \"2b0e07f4-337a-4961-aa14-36f1c8989433\" (UID: \"2b0e07f4-337a-4961-aa14-36f1c8989433\") " Oct 30 22:01:49.099553 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:49.099430 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b0e07f4-337a-4961-aa14-36f1c8989433-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b0e07f4-337a-4961-aa14-36f1c8989433-maas-api-openshift-ca\") pod \"2b0e07f4-337a-4961-aa14-36f1c8989433\" (UID: \"2b0e07f4-337a-4961-aa14-36f1c8989433\") " Oct 30 22:01:49.099650 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:49.099607 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11aa0088-fac9-4ff4-a925-e241e613137f-kube-api-access-7mrzv\" (UniqueName: \"kubernetes.io/projected/11aa0088-fac9-4ff4-a925-e241e613137f-kube-api-access-7mrzv\") pod \"11aa0088-fac9-4ff4-a925-e241e613137f\" (UID: \"11aa0088-fac9-4ff4-a925-e241e613137f\") " Oct 30 22:01:49.099650 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:49.099643 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11aa0088-fac9-4ff4-a925-e241e613137f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11aa0088-fac9-4ff4-a925-e241e613137f-maas-api-openshift-ca\") pod \"11aa0088-fac9-4ff4-a925-e241e613137f\" (UID: \"11aa0088-fac9-4ff4-a925-e241e613137f\") " Oct 30 22:01:49.099746 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:49.099668 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b0e07f4-337a-4961-aa14-36f1c8989433-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b0e07f4-337a-4961-aa14-36f1c8989433-maas-api-xks-ca\") pod \"2b0e07f4-337a-4961-aa14-36f1c8989433\" (UID: \"2b0e07f4-337a-4961-aa14-36f1c8989433\") " Oct 30 22:01:49.099918 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:49.099892 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b0e07f4-337a-4961-aa14-36f1c8989433-maas-api-openshift-ca" pod "2b0e07f4-337a-4961-aa14-36f1c8989433" (UID: "2b0e07f4-337a-4961-aa14-36f1c8989433"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:01:49.100208 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:49.100186 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b0e07f4-337a-4961-aa14-36f1c8989433-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:49.100271 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:49.100251 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11aa0088-fac9-4ff4-a925-e241e613137f-maas-api-openshift-ca" pod "11aa0088-fac9-4ff4-a925-e241e613137f" (UID: "11aa0088-fac9-4ff4-a925-e241e613137f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:01:49.102100 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:49.102078 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11aa0088-fac9-4ff4-a925-e241e613137f-maas-api-xks-ca" pod "11aa0088-fac9-4ff4-a925-e241e613137f" (UID: "11aa0088-fac9-4ff4-a925-e241e613137f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:01:49.102240 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:49.102187 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b0e07f4-337a-4961-aa14-36f1c8989433-kube-api-access-gzscv" pod "2b0e07f4-337a-4961-aa14-36f1c8989433" (UID: "2b0e07f4-337a-4961-aa14-36f1c8989433"). InnerVolumeSpecName "kube-api-access-gzscv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:49.102518 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:49.102490 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11aa0088-fac9-4ff4-a925-e241e613137f-kube-api-access-7mrzv" pod "11aa0088-fac9-4ff4-a925-e241e613137f" (UID: "11aa0088-fac9-4ff4-a925-e241e613137f"). InnerVolumeSpecName "kube-api-access-7mrzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:49.102735 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:49.102703 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b0e07f4-337a-4961-aa14-36f1c8989433-maas-api-xks-ca" pod "2b0e07f4-337a-4961-aa14-36f1c8989433" (UID: "2b0e07f4-337a-4961-aa14-36f1c8989433"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:01:49.201885 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:49.201808 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mrzv\" (UniqueName: \"kubernetes.io/projected/11aa0088-fac9-4ff4-a925-e241e613137f-kube-api-access-7mrzv\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:49.201885 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:49.201839 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11aa0088-fac9-4ff4-a925-e241e613137f-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:49.201885 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:49.201849 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b0e07f4-337a-4961-aa14-36f1c8989433-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:49.201885 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:49.201861 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11aa0088-fac9-4ff4-a925-e241e613137f-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:49.201885 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:49.201870 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzscv\" (UniqueName: \"kubernetes.io/projected/2b0e07f4-337a-4961-aa14-36f1c8989433-kube-api-access-gzscv\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:49.685450 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:01:49.685413 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:01:49.867874 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:49.867790 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-816ca17c-l2249" event={"ID":"11aa0088-fac9-4ff4-a925-e241e613137f","Type":"ContainerDied","Data":"56118db5d6d1421826b3c23a81315c35e8b355608caab65458faaf9e3d02b3c6"} Oct 30 22:01:49.868304 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:49.867808 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-816ca17c-l2249" Oct 30 22:01:49.868304 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:49.867902 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56118db5d6d1421826b3c23a81315c35e8b355608caab65458faaf9e3d02b3c6" Oct 30 22:01:49.869305 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:49.869281 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe1cd9b-dxgzt" event={"ID":"2b0e07f4-337a-4961-aa14-36f1c8989433","Type":"ContainerDied","Data":"0fffd5cd4bcd81c9ab8c10282699e1623e7b58858ce7efebdde883e6332e1b1c"} Oct 30 22:01:49.869439 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:49.869328 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fffd5cd4bcd81c9ab8c10282699e1623e7b58858ce7efebdde883e6332e1b1c" Oct 30 22:01:49.869559 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:49.869541 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe1cd9b-dxgzt" Oct 30 22:01:50.043704 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:50.043679 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c34bf965-stnn7" Oct 30 22:01:50.063811 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:50.063789 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd387d1e-lpzbb" Oct 30 22:01:50.113707 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:50.113676 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ee952bc-3471-4387-8b50-b242f8dc9b44-kube-api-access-4h6ds\" (UniqueName: \"kubernetes.io/projected/5ee952bc-3471-4387-8b50-b242f8dc9b44-kube-api-access-4h6ds\") pod \"5ee952bc-3471-4387-8b50-b242f8dc9b44\" (UID: \"5ee952bc-3471-4387-8b50-b242f8dc9b44\") " Oct 30 22:01:50.113878 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:50.113733 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/282c407f-8be5-4c81-9d4c-e67f7ba44cc9-kube-api-access-zq9qm\" (UniqueName: \"kubernetes.io/projected/282c407f-8be5-4c81-9d4c-e67f7ba44cc9-kube-api-access-zq9qm\") pod \"282c407f-8be5-4c81-9d4c-e67f7ba44cc9\" (UID: \"282c407f-8be5-4c81-9d4c-e67f7ba44cc9\") " Oct 30 22:01:50.113878 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:50.113807 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/282c407f-8be5-4c81-9d4c-e67f7ba44cc9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/282c407f-8be5-4c81-9d4c-e67f7ba44cc9-maas-api-openshift-ca\") pod \"282c407f-8be5-4c81-9d4c-e67f7ba44cc9\" (UID: \"282c407f-8be5-4c81-9d4c-e67f7ba44cc9\") " Oct 30 22:01:50.113878 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:50.113845 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ee952bc-3471-4387-8b50-b242f8dc9b44-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ee952bc-3471-4387-8b50-b242f8dc9b44-maas-api-xks-ca\") pod \"5ee952bc-3471-4387-8b50-b242f8dc9b44\" (UID: \"5ee952bc-3471-4387-8b50-b242f8dc9b44\") " Oct 30 22:01:50.114058 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:50.113886 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ee952bc-3471-4387-8b50-b242f8dc9b44-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ee952bc-3471-4387-8b50-b242f8dc9b44-maas-api-openshift-ca\") pod \"5ee952bc-3471-4387-8b50-b242f8dc9b44\" (UID: \"5ee952bc-3471-4387-8b50-b242f8dc9b44\") " Oct 30 22:01:50.114058 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:50.113920 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/282c407f-8be5-4c81-9d4c-e67f7ba44cc9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/282c407f-8be5-4c81-9d4c-e67f7ba44cc9-maas-api-xks-ca\") pod \"282c407f-8be5-4c81-9d4c-e67f7ba44cc9\" (UID: \"282c407f-8be5-4c81-9d4c-e67f7ba44cc9\") " Oct 30 22:01:50.114240 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:50.114214 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/282c407f-8be5-4c81-9d4c-e67f7ba44cc9-maas-api-openshift-ca" pod "282c407f-8be5-4c81-9d4c-e67f7ba44cc9" (UID: "282c407f-8be5-4c81-9d4c-e67f7ba44cc9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:01:50.114363 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:50.114272 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ee952bc-3471-4387-8b50-b242f8dc9b44-maas-api-openshift-ca" pod "5ee952bc-3471-4387-8b50-b242f8dc9b44" (UID: "5ee952bc-3471-4387-8b50-b242f8dc9b44"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:01:50.114534 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:50.114511 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/282c407f-8be5-4c81-9d4c-e67f7ba44cc9-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:50.114617 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:50.114538 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ee952bc-3471-4387-8b50-b242f8dc9b44-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:50.116034 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:50.116004 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/282c407f-8be5-4c81-9d4c-e67f7ba44cc9-kube-api-access-zq9qm" pod "282c407f-8be5-4c81-9d4c-e67f7ba44cc9" (UID: "282c407f-8be5-4c81-9d4c-e67f7ba44cc9"). InnerVolumeSpecName "kube-api-access-zq9qm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:50.116322 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:50.116302 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ee952bc-3471-4387-8b50-b242f8dc9b44-kube-api-access-4h6ds" pod "5ee952bc-3471-4387-8b50-b242f8dc9b44" (UID: "5ee952bc-3471-4387-8b50-b242f8dc9b44"). InnerVolumeSpecName "kube-api-access-4h6ds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:50.116404 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:50.116323 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/282c407f-8be5-4c81-9d4c-e67f7ba44cc9-maas-api-xks-ca" pod "282c407f-8be5-4c81-9d4c-e67f7ba44cc9" (UID: "282c407f-8be5-4c81-9d4c-e67f7ba44cc9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:01:50.116404 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:50.116301 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ee952bc-3471-4387-8b50-b242f8dc9b44-maas-api-xks-ca" pod "5ee952bc-3471-4387-8b50-b242f8dc9b44" (UID: "5ee952bc-3471-4387-8b50-b242f8dc9b44"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:01:50.216180 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:50.216110 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h6ds\" (UniqueName: \"kubernetes.io/projected/5ee952bc-3471-4387-8b50-b242f8dc9b44-kube-api-access-4h6ds\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:50.216180 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:50.216139 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq9qm\" (UniqueName: \"kubernetes.io/projected/282c407f-8be5-4c81-9d4c-e67f7ba44cc9-kube-api-access-zq9qm\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:50.216180 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:50.216149 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ee952bc-3471-4387-8b50-b242f8dc9b44-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:50.216180 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:50.216158 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/282c407f-8be5-4c81-9d4c-e67f7ba44cc9-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:50.876591 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:50.876512 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c34bf965-stnn7" Oct 30 22:01:50.876591 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:50.876516 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c34bf965-stnn7" event={"ID":"5ee952bc-3471-4387-8b50-b242f8dc9b44","Type":"ContainerDied","Data":"b5d5cf1e3273cac89cfae295fc83116b55f8268c3cef21b54251f7c139367e31"} Oct 30 22:01:50.876591 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:50.876572 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5d5cf1e3273cac89cfae295fc83116b55f8268c3cef21b54251f7c139367e31" Oct 30 22:01:50.878084 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:50.878059 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd387d1e-lpzbb" Oct 30 22:01:50.878192 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:50.878095 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd387d1e-lpzbb" event={"ID":"282c407f-8be5-4c81-9d4c-e67f7ba44cc9","Type":"ContainerDied","Data":"72659e7c79cff47ae5c1fbb6753fed38c0764ab9604a06cebe5e5218380f18ae"} Oct 30 22:01:50.878192 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:50.878131 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72659e7c79cff47ae5c1fbb6753fed38c0764ab9604a06cebe5e5218380f18ae" Oct 30 22:01:51.885942 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:51.885909 4560 generic.go:356] "Generic (PLEG): container finished" podID="7f4405ac-93f4-4a1d-a335-5990238c4294" containerID="de6e083df99964d0fec1de18f4a318649336521e9d7ad8b20d61d0c3d1dc1c10" exitCode=0 Oct 30 22:01:51.886471 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:51.885977 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" event={"ID":"7f4405ac-93f4-4a1d-a335-5990238c4294","Type":"ContainerDied","Data":"de6e083df99964d0fec1de18f4a318649336521e9d7ad8b20d61d0c3d1dc1c10"} Oct 30 22:01:51.886471 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:51.886388 4560 scope.go:122] "RemoveContainer" containerID="de6e083df99964d0fec1de18f4a318649336521e9d7ad8b20d61d0c3d1dc1c10" Oct 30 22:01:51.887787 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:51.887757 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:01:51.887890 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:51.887842 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:01:51.887955 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:51.887892 4560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:01:52.893619 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:52.893587 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" event={"ID":"7f4405ac-93f4-4a1d-a335-5990238c4294","Type":"ContainerStarted","Data":"feb158cd0379b553dd6379400a6696ab4eaa69c5cc34a49485737fbb7af15554"} Oct 30 22:01:53.815112 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:53.815066 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" Oct 30 22:01:53.815750 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:53.815721 4560 patch_prober.go:29] interesting pod/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 30 22:01:53.815816 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:53.815779 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" podUID="7f4405ac-93f4-4a1d-a335-5990238c4294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 30 22:01:54.815646 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:54.815608 4560 patch_prober.go:29] interesting pod/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 30 22:01:54.816145 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:54.815671 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" podUID="7f4405ac-93f4-4a1d-a335-5990238c4294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 30 22:01:55.274364 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:55.274320 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj"] Oct 30 22:01:55.274590 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:55.274554 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" podUID="64e8e251-1ae0-42cc-b46b-e8f11b3395ff" containerName="istio-proxy" containerID="cri-o://d76a6c9fed686429cada32d362b582bfc579ba7fa56470957969ae5570e971cb" gracePeriod=30 Oct 30 22:01:55.815812 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:55.815778 4560 patch_prober.go:29] interesting pod/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 30 22:01:55.816214 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:55.815836 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" podUID="7f4405ac-93f4-4a1d-a335-5990238c4294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 30 22:01:56.367665 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:56.367627 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a2dced-pbwv6"] Oct 30 22:01:56.368511 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:56.368487 4560 cpu_manager.go:414] "containerMap: removing container" podUID="90966441-80bf-4bc5-8e68-4dea00d9f779" containerName="main" Oct 30 22:01:56.368650 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:56.368638 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90966441-80bf-4bc5-8e68-4dea00d9f779" containerName="main" Oct 30 22:01:56.368730 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:56.368721 4560 cpu_manager.go:414] "containerMap: removing container" podUID="2b0e07f4-337a-4961-aa14-36f1c8989433" containerName="revoke-keys" Oct 30 22:01:56.368797 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:56.368787 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b0e07f4-337a-4961-aa14-36f1c8989433" containerName="revoke-keys" Oct 30 22:01:56.368879 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:56.368870 4560 cpu_manager.go:414] "containerMap: removing container" podUID="11aa0088-fac9-4ff4-a925-e241e613137f" containerName="revoke-keys" Oct 30 22:01:56.368946 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:56.368938 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11aa0088-fac9-4ff4-a925-e241e613137f" containerName="revoke-keys" Oct 30 22:01:56.369019 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:56.369011 4560 cpu_manager.go:414] "containerMap: removing container" podUID="5ee952bc-3471-4387-8b50-b242f8dc9b44" containerName="revoke-keys" Oct 30 22:01:56.369086 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:56.369077 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ee952bc-3471-4387-8b50-b242f8dc9b44" containerName="revoke-keys" Oct 30 22:01:56.369166 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:56.369158 4560 cpu_manager.go:414] "containerMap: removing container" podUID="282c407f-8be5-4c81-9d4c-e67f7ba44cc9" containerName="revoke-keys" Oct 30 22:01:56.369231 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:56.369223 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="282c407f-8be5-4c81-9d4c-e67f7ba44cc9" containerName="revoke-keys" Oct 30 22:01:56.369434 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:56.369423 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="282c407f-8be5-4c81-9d4c-e67f7ba44cc9" containerName="revoke-keys" Oct 30 22:01:56.369525 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:56.369516 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="90966441-80bf-4bc5-8e68-4dea00d9f779" containerName="main" Oct 30 22:01:56.369592 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:56.369584 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b0e07f4-337a-4961-aa14-36f1c8989433" containerName="revoke-keys" Oct 30 22:01:56.369655 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:56.369647 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="11aa0088-fac9-4ff4-a925-e241e613137f" containerName="revoke-keys" Oct 30 22:01:56.369721 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:56.369713 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ee952bc-3471-4387-8b50-b242f8dc9b44" containerName="revoke-keys" Oct 30 22:01:56.383942 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:56.383915 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a2dced-pbwv6" Oct 30 22:01:56.455715 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:56.455673 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:01:56.456084 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:56.456064 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:01:56.457093 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:56.457062 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a2dced-pbwv6"] Oct 30 22:01:56.472195 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:56.472165 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:01:56.485467 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:56.485443 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:01:56.518156 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:56.518127 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636e1507-92bf-48ea-b1c5-b93bc4d08f66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a2dced-pbwv6\" (UID: \"636e1507-92bf-48ea-b1c5-b93bc4d08f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a2dced-pbwv6" Oct 30 22:01:56.518332 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:56.518276 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636e1507-92bf-48ea-b1c5-b93bc4d08f66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a2dced-pbwv6\" (UID: \"636e1507-92bf-48ea-b1c5-b93bc4d08f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a2dced-pbwv6" Oct 30 22:01:56.518461 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:56.518435 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tvdp\" (UniqueName: \"kubernetes.io/projected/636e1507-92bf-48ea-b1c5-b93bc4d08f66-kube-api-access-5tvdp\") pod \"maas-api-revoke-keys-e2e-worker-w6-a2dced-pbwv6\" (UID: \"636e1507-92bf-48ea-b1c5-b93bc4d08f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a2dced-pbwv6" Oct 30 22:01:56.619286 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:56.619205 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tvdp\" (UniqueName: \"kubernetes.io/projected/636e1507-92bf-48ea-b1c5-b93bc4d08f66-kube-api-access-5tvdp\") pod \"maas-api-revoke-keys-e2e-worker-w6-a2dced-pbwv6\" (UID: \"636e1507-92bf-48ea-b1c5-b93bc4d08f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a2dced-pbwv6" Oct 30 22:01:56.619286 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:56.619281 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636e1507-92bf-48ea-b1c5-b93bc4d08f66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a2dced-pbwv6\" (UID: \"636e1507-92bf-48ea-b1c5-b93bc4d08f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a2dced-pbwv6" Oct 30 22:01:56.619546 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:56.619402 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636e1507-92bf-48ea-b1c5-b93bc4d08f66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a2dced-pbwv6\" (UID: \"636e1507-92bf-48ea-b1c5-b93bc4d08f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a2dced-pbwv6" Oct 30 22:01:56.620123 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:56.620096 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636e1507-92bf-48ea-b1c5-b93bc4d08f66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a2dced-pbwv6\" (UID: \"636e1507-92bf-48ea-b1c5-b93bc4d08f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a2dced-pbwv6" Oct 30 22:01:56.621802 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:56.621773 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636e1507-92bf-48ea-b1c5-b93bc4d08f66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a2dced-pbwv6\" (UID: \"636e1507-92bf-48ea-b1c5-b93bc4d08f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a2dced-pbwv6" Oct 30 22:01:56.678400 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:56.678332 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tvdp\" (UniqueName: \"kubernetes.io/projected/636e1507-92bf-48ea-b1c5-b93bc4d08f66-kube-api-access-5tvdp\") pod \"maas-api-revoke-keys-e2e-worker-w6-a2dced-pbwv6\" (UID: \"636e1507-92bf-48ea-b1c5-b93bc4d08f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a2dced-pbwv6" Oct 30 22:01:56.700285 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:56.700256 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a2dced-pbwv6" Oct 30 22:01:56.815572 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:56.815394 4560 patch_prober.go:29] interesting pod/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 30 22:01:56.815572 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:56.815474 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" podUID="7f4405ac-93f4-4a1d-a335-5990238c4294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 30 22:01:57.104975 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:57.104933 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a2dced-pbwv6"] Oct 30 22:01:57.112049 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:01:57.112022 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod636e1507_92bf_48ea_b1c5_b93bc4d08f66.slice/crio-2c12a8c3061162a6dc7e5144e9c534ed22217116c020831a80c8dcc18b265fa8 WatchSource:0}: Error finding container 2c12a8c3061162a6dc7e5144e9c534ed22217116c020831a80c8dcc18b265fa8: Status 404 returned error can't find the container with id 2c12a8c3061162a6dc7e5144e9c534ed22217116c020831a80c8dcc18b265fa8 Oct 30 22:01:57.815136 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:57.815098 4560 patch_prober.go:29] interesting pod/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 30 22:01:57.815314 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:57.815151 4560 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" podUID="7f4405ac-93f4-4a1d-a335-5990238c4294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 30 22:01:57.924771 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:57.924732 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a2dced-pbwv6" event={"ID":"636e1507-92bf-48ea-b1c5-b93bc4d08f66","Type":"ContainerStarted","Data":"f41d81830e6ac88bd13279fd47b762e4a0c3f0393689e6f22d7dbc403adf4671"} Oct 30 22:01:57.924949 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:57.924789 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a2dced-pbwv6" event={"ID":"636e1507-92bf-48ea-b1c5-b93bc4d08f66","Type":"ContainerStarted","Data":"2c12a8c3061162a6dc7e5144e9c534ed22217116c020831a80c8dcc18b265fa8"} Oct 30 22:01:58.068989 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:58.068873 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a2dced-pbwv6" podStartSLOduration=2.068854327 podStartE2EDuration="2.068854327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:58.064261504 +0000 UTC m=+2893.618961277" watchObservedRunningTime="2026-10-30 22:01:58.068854327 +0000 UTC m=+2893.623554105" Oct 30 22:01:58.792407 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:58.791481 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" Oct 30 22:01:58.839689 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:58.839658 4560 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" Oct 30 22:01:58.953814 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:58.953724 4560 generic.go:356] "Generic (PLEG): container finished" podID="636e1507-92bf-48ea-b1c5-b93bc4d08f66" containerID="f41d81830e6ac88bd13279fd47b762e4a0c3f0393689e6f22d7dbc403adf4671" exitCode=0 Oct 30 22:01:58.955078 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:58.955048 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a2dced-pbwv6" event={"ID":"636e1507-92bf-48ea-b1c5-b93bc4d08f66","Type":"ContainerDied","Data":"f41d81830e6ac88bd13279fd47b762e4a0c3f0393689e6f22d7dbc403adf4671"} Oct 30 22:01:58.955981 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:58.955958 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" Oct 30 22:01:59.412767 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:59.412730 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ca0a55-7ww8s"] Oct 30 22:01:59.432405 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:01:59.432374 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ca0a55-7ww8s"] Oct 30 22:02:00.179304 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.179279 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a2dced-pbwv6" Oct 30 22:02:00.261029 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.260997 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/636e1507-92bf-48ea-b1c5-b93bc4d08f66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636e1507-92bf-48ea-b1c5-b93bc4d08f66-maas-api-xks-ca\") pod \"636e1507-92bf-48ea-b1c5-b93bc4d08f66\" (UID: \"636e1507-92bf-48ea-b1c5-b93bc4d08f66\") " Oct 30 22:02:00.261198 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.261034 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/636e1507-92bf-48ea-b1c5-b93bc4d08f66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636e1507-92bf-48ea-b1c5-b93bc4d08f66-maas-api-openshift-ca\") pod \"636e1507-92bf-48ea-b1c5-b93bc4d08f66\" (UID: \"636e1507-92bf-48ea-b1c5-b93bc4d08f66\") " Oct 30 22:02:00.261198 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.261066 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/636e1507-92bf-48ea-b1c5-b93bc4d08f66-kube-api-access-5tvdp\" (UniqueName: \"kubernetes.io/projected/636e1507-92bf-48ea-b1c5-b93bc4d08f66-kube-api-access-5tvdp\") pod \"636e1507-92bf-48ea-b1c5-b93bc4d08f66\" (UID: \"636e1507-92bf-48ea-b1c5-b93bc4d08f66\") " Oct 30 22:02:00.261465 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.261441 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/636e1507-92bf-48ea-b1c5-b93bc4d08f66-maas-api-openshift-ca" pod "636e1507-92bf-48ea-b1c5-b93bc4d08f66" (UID: "636e1507-92bf-48ea-b1c5-b93bc4d08f66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:00.261566 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.261510 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636e1507-92bf-48ea-b1c5-b93bc4d08f66-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:00.263255 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.263228 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/636e1507-92bf-48ea-b1c5-b93bc4d08f66-maas-api-xks-ca" pod "636e1507-92bf-48ea-b1c5-b93bc4d08f66" (UID: "636e1507-92bf-48ea-b1c5-b93bc4d08f66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:00.263404 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.263385 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/636e1507-92bf-48ea-b1c5-b93bc4d08f66-kube-api-access-5tvdp" pod "636e1507-92bf-48ea-b1c5-b93bc4d08f66" (UID: "636e1507-92bf-48ea-b1c5-b93bc4d08f66"). InnerVolumeSpecName "kube-api-access-5tvdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:00.362244 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.362211 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636e1507-92bf-48ea-b1c5-b93bc4d08f66-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:00.362244 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.362240 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tvdp\" (UniqueName: \"kubernetes.io/projected/636e1507-92bf-48ea-b1c5-b93bc4d08f66-kube-api-access-5tvdp\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:00.520822 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.520796 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" Oct 30 22:02:00.563611 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.563585 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-istio-token\" (UniqueName: \"kubernetes.io/projected/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-istio-token\") pod \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\" (UID: \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\") " Oct 30 22:02:00.563752 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.563632 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-workload-certs\") pod \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\" (UID: \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\") " Oct 30 22:02:00.563752 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.563680 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-istio-podinfo\") pod \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\" (UID: \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\") " Oct 30 22:02:00.563752 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.563709 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-workload-socket\") pod \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\" (UID: \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\") " Oct 30 22:02:00.563752 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.563731 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-kube-api-access-km9v9\" (UniqueName: \"kubernetes.io/projected/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-kube-api-access-km9v9\") pod \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\" (UID: \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\") " Oct 30 22:02:00.563949 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.563799 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-credential-socket\") pod \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\" (UID: \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\") " Oct 30 22:02:00.563949 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.563839 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-istiod-ca-cert\") pod \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\" (UID: \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\") " Oct 30 22:02:00.563949 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.563866 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-istio-envoy\") pod \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\" (UID: \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\") " Oct 30 22:02:00.564126 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.564002 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-workload-certs" pod "64e8e251-1ae0-42cc-b46b-e8f11b3395ff" (UID: "64e8e251-1ae0-42cc-b46b-e8f11b3395ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:00.564126 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.564007 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-workload-socket" pod "64e8e251-1ae0-42cc-b46b-e8f11b3395ff" (UID: "64e8e251-1ae0-42cc-b46b-e8f11b3395ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:00.564126 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.564111 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-credential-socket" pod "64e8e251-1ae0-42cc-b46b-e8f11b3395ff" (UID: "64e8e251-1ae0-42cc-b46b-e8f11b3395ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:00.564317 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.564290 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-istiod-ca-cert" pod "64e8e251-1ae0-42cc-b46b-e8f11b3395ff" (UID: "64e8e251-1ae0-42cc-b46b-e8f11b3395ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:00.564479 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.564463 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-workload-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:00.564546 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.564485 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-credential-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:00.564546 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.564495 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-istiod-ca-cert\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:00.564546 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.564505 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-workload-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:00.565797 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.565776 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-istio-token" pod "64e8e251-1ae0-42cc-b46b-e8f11b3395ff" (UID: "64e8e251-1ae0-42cc-b46b-e8f11b3395ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:00.565886 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.565834 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-kube-api-access-km9v9" pod "64e8e251-1ae0-42cc-b46b-e8f11b3395ff" (UID: "64e8e251-1ae0-42cc-b46b-e8f11b3395ff"). InnerVolumeSpecName "kube-api-access-km9v9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:00.566300 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.566272 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-istio-podinfo" pod "64e8e251-1ae0-42cc-b46b-e8f11b3395ff" (UID: "64e8e251-1ae0-42cc-b46b-e8f11b3395ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:02:00.566300 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.566272 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-istio-envoy" pod "64e8e251-1ae0-42cc-b46b-e8f11b3395ff" (UID: "64e8e251-1ae0-42cc-b46b-e8f11b3395ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:00.576884 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.576862 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f12246a4-1d55-475b-bcdb-5472da93f641" path="/var/lib/kubelet/pods/f12246a4-1d55-475b-bcdb-5472da93f641/volumes" Oct 30 22:02:00.665662 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.665627 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-istio-data\") pod \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\" (UID: \"64e8e251-1ae0-42cc-b46b-e8f11b3395ff\") " Oct 30 22:02:00.665917 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.665892 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-istio-data" pod "64e8e251-1ae0-42cc-b46b-e8f11b3395ff" (UID: "64e8e251-1ae0-42cc-b46b-e8f11b3395ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:00.666058 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.666040 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-istio-data\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:00.666100 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.666065 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-istio-token\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:00.666100 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.666079 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-istio-podinfo\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:00.666100 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.666095 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km9v9\" (UniqueName: \"kubernetes.io/projected/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-kube-api-access-km9v9\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:00.666199 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.666109 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64e8e251-1ae0-42cc-b46b-e8f11b3395ff-istio-envoy\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:00.809231 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.809143 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9"] Oct 30 22:02:00.826766 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.826739 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p"] Oct 30 22:02:00.847162 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.847133 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q"] Oct 30 22:02:00.974871 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.969374 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a2dced-pbwv6" event={"ID":"636e1507-92bf-48ea-b1c5-b93bc4d08f66","Type":"ContainerDied","Data":"2c12a8c3061162a6dc7e5144e9c534ed22217116c020831a80c8dcc18b265fa8"} Oct 30 22:02:00.974871 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.969438 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c12a8c3061162a6dc7e5144e9c534ed22217116c020831a80c8dcc18b265fa8" Oct 30 22:02:00.974871 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.969390 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a2dced-pbwv6" Oct 30 22:02:00.974871 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.971710 4560 generic.go:356] "Generic (PLEG): container finished" podID="64e8e251-1ae0-42cc-b46b-e8f11b3395ff" containerID="d76a6c9fed686429cada32d362b582bfc579ba7fa56470957969ae5570e971cb" exitCode=0 Oct 30 22:02:00.974871 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.971756 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" event={"ID":"64e8e251-1ae0-42cc-b46b-e8f11b3395ff","Type":"ContainerDied","Data":"d76a6c9fed686429cada32d362b582bfc579ba7fa56470957969ae5570e971cb"} Oct 30 22:02:00.974871 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.971790 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" event={"ID":"64e8e251-1ae0-42cc-b46b-e8f11b3395ff","Type":"ContainerDied","Data":"9418b2301eabdc205ed9076c0467f099fec9edac673815698d4e5f547e6264a5"} Oct 30 22:02:00.974871 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.971819 4560 scope.go:122] "RemoveContainer" containerID="d76a6c9fed686429cada32d362b582bfc579ba7fa56470957969ae5570e971cb" Oct 30 22:02:00.974871 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.971979 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj" Oct 30 22:02:00.986840 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.986819 4560 scope.go:122] "RemoveContainer" containerID="601da35e07d40b926692b97ec4f18d9ce1169c75d199608c6fb69fb27769b567" Oct 30 22:02:00.995471 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.995453 4560 scope.go:122] "RemoveContainer" containerID="d76a6c9fed686429cada32d362b582bfc579ba7fa56470957969ae5570e971cb" Oct 30 22:02:00.995747 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:02:00.995725 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d76a6c9fed686429cada32d362b582bfc579ba7fa56470957969ae5570e971cb\": container with ID starting with d76a6c9fed686429cada32d362b582bfc579ba7fa56470957969ae5570e971cb not found: ID does not exist" containerID="d76a6c9fed686429cada32d362b582bfc579ba7fa56470957969ae5570e971cb" Oct 30 22:02:00.995802 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.995756 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d76a6c9fed686429cada32d362b582bfc579ba7fa56470957969ae5570e971cb"} err="failed to get container status \"d76a6c9fed686429cada32d362b582bfc579ba7fa56470957969ae5570e971cb\": rpc error: code = NotFound desc = could not find container \"d76a6c9fed686429cada32d362b582bfc579ba7fa56470957969ae5570e971cb\": container with ID starting with d76a6c9fed686429cada32d362b582bfc579ba7fa56470957969ae5570e971cb not found: ID does not exist" Oct 30 22:02:00.995802 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.995774 4560 scope.go:122] "RemoveContainer" containerID="601da35e07d40b926692b97ec4f18d9ce1169c75d199608c6fb69fb27769b567" Oct 30 22:02:00.996001 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:02:00.995983 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"601da35e07d40b926692b97ec4f18d9ce1169c75d199608c6fb69fb27769b567\": container with ID starting with 601da35e07d40b926692b97ec4f18d9ce1169c75d199608c6fb69fb27769b567 not found: ID does not exist" containerID="601da35e07d40b926692b97ec4f18d9ce1169c75d199608c6fb69fb27769b567" Oct 30 22:02:00.996064 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:00.996006 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"601da35e07d40b926692b97ec4f18d9ce1169c75d199608c6fb69fb27769b567"} err="failed to get container status \"601da35e07d40b926692b97ec4f18d9ce1169c75d199608c6fb69fb27769b567\": rpc error: code = NotFound desc = could not find container \"601da35e07d40b926692b97ec4f18d9ce1169c75d199608c6fb69fb27769b567\": container with ID starting with 601da35e07d40b926692b97ec4f18d9ce1169c75d199608c6fb69fb27769b567 not found: ID does not exist" Oct 30 22:02:01.271619 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:01.271580 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj"] Oct 30 22:02:01.277395 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:01.277367 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c4e874-openshift-default-7848748d5c-bsqpj"] Oct 30 22:02:02.578554 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:02.578510 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64e8e251-1ae0-42cc-b46b-e8f11b3395ff" path="/var/lib/kubelet/pods/64e8e251-1ae0-42cc-b46b-e8f11b3395ff/volumes" Oct 30 22:02:07.547127 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:07.547080 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a2dced-7f7d56769b-9fh9l"] Oct 30 22:02:09.630789 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:09.630755 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-991335-dplhn"] Oct 30 22:02:09.631200 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:09.631157 4560 cpu_manager.go:414] "containerMap: removing container" podUID="64e8e251-1ae0-42cc-b46b-e8f11b3395ff" containerName="istio-proxy" Oct 30 22:02:09.631200 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:09.631167 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64e8e251-1ae0-42cc-b46b-e8f11b3395ff" containerName="istio-proxy" Oct 30 22:02:09.631200 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:09.631181 4560 cpu_manager.go:414] "containerMap: removing container" podUID="64e8e251-1ae0-42cc-b46b-e8f11b3395ff" containerName="istio-proxy" Oct 30 22:02:09.631200 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:09.631187 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64e8e251-1ae0-42cc-b46b-e8f11b3395ff" containerName="istio-proxy" Oct 30 22:02:09.631349 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:09.631210 4560 cpu_manager.go:414] "containerMap: removing container" podUID="636e1507-92bf-48ea-b1c5-b93bc4d08f66" containerName="revoke-keys" Oct 30 22:02:09.631349 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:09.631215 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="636e1507-92bf-48ea-b1c5-b93bc4d08f66" containerName="revoke-keys" Oct 30 22:02:09.631349 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:09.631279 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="64e8e251-1ae0-42cc-b46b-e8f11b3395ff" containerName="istio-proxy" Oct 30 22:02:09.631349 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:09.631286 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="64e8e251-1ae0-42cc-b46b-e8f11b3395ff" containerName="istio-proxy" Oct 30 22:02:09.631349 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:09.631292 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="636e1507-92bf-48ea-b1c5-b93bc4d08f66" containerName="revoke-keys" Oct 30 22:02:09.636412 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:09.636390 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-991335-dplhn" Oct 30 22:02:09.689903 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:09.689866 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-991335-dplhn"] Oct 30 22:02:09.695239 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:09.695208 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr6pd\" (UniqueName: \"kubernetes.io/projected/404224b2-3856-49b0-a6fe-f28335f2580c-kube-api-access-pr6pd\") pod \"maas-api-revoke-keys-e2e-worker-w5-991335-dplhn\" (UID: \"404224b2-3856-49b0-a6fe-f28335f2580c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-991335-dplhn" Oct 30 22:02:09.695417 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:09.695321 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/404224b2-3856-49b0-a6fe-f28335f2580c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-991335-dplhn\" (UID: \"404224b2-3856-49b0-a6fe-f28335f2580c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-991335-dplhn" Oct 30 22:02:09.695470 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:09.695458 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/404224b2-3856-49b0-a6fe-f28335f2580c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-991335-dplhn\" (UID: \"404224b2-3856-49b0-a6fe-f28335f2580c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-991335-dplhn" Oct 30 22:02:09.752593 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:09.752535 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:02:09.757831 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:09.752866 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:02:09.761111 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:09.758176 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:02:09.761681 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:09.761658 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:02:09.797109 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:09.797079 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr6pd\" (UniqueName: \"kubernetes.io/projected/404224b2-3856-49b0-a6fe-f28335f2580c-kube-api-access-pr6pd\") pod \"maas-api-revoke-keys-e2e-worker-w5-991335-dplhn\" (UID: \"404224b2-3856-49b0-a6fe-f28335f2580c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-991335-dplhn" Oct 30 22:02:09.797288 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:09.797147 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/404224b2-3856-49b0-a6fe-f28335f2580c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-991335-dplhn\" (UID: \"404224b2-3856-49b0-a6fe-f28335f2580c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-991335-dplhn" Oct 30 22:02:09.797288 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:09.797187 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/404224b2-3856-49b0-a6fe-f28335f2580c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-991335-dplhn\" (UID: \"404224b2-3856-49b0-a6fe-f28335f2580c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-991335-dplhn" Oct 30 22:02:09.797881 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:09.797859 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/404224b2-3856-49b0-a6fe-f28335f2580c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-991335-dplhn\" (UID: \"404224b2-3856-49b0-a6fe-f28335f2580c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-991335-dplhn" Oct 30 22:02:09.799656 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:09.799631 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/404224b2-3856-49b0-a6fe-f28335f2580c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-991335-dplhn\" (UID: \"404224b2-3856-49b0-a6fe-f28335f2580c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-991335-dplhn" Oct 30 22:02:09.855888 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:09.855858 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr6pd\" (UniqueName: \"kubernetes.io/projected/404224b2-3856-49b0-a6fe-f28335f2580c-kube-api-access-pr6pd\") pod \"maas-api-revoke-keys-e2e-worker-w5-991335-dplhn\" (UID: \"404224b2-3856-49b0-a6fe-f28335f2580c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-991335-dplhn" Oct 30 22:02:09.949739 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:09.949646 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-991335-dplhn" Oct 30 22:02:10.324571 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:10.324533 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-991335-dplhn"] Oct 30 22:02:10.329866 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:02:10.329835 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod404224b2_3856_49b0_a6fe_f28335f2580c.slice/crio-b0364a5feaea848ebf1952fa3284ed1cb3b2265468f8e18c373ea049e1b69707 WatchSource:0}: Error finding container b0364a5feaea848ebf1952fa3284ed1cb3b2265468f8e18c373ea049e1b69707: Status 404 returned error can't find the container with id b0364a5feaea848ebf1952fa3284ed1cb3b2265468f8e18c373ea049e1b69707 Oct 30 22:02:11.054911 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:11.054874 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-991335-dplhn" event={"ID":"404224b2-3856-49b0-a6fe-f28335f2580c","Type":"ContainerStarted","Data":"3b252fd81d0867b3d3cb86e692a3b1ea3e7e4615a7d1bd5a4178b443d8411ebe"} Oct 30 22:02:11.055325 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:11.054938 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-991335-dplhn" event={"ID":"404224b2-3856-49b0-a6fe-f28335f2580c","Type":"ContainerStarted","Data":"b0364a5feaea848ebf1952fa3284ed1cb3b2265468f8e18c373ea049e1b69707"} Oct 30 22:02:11.167222 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:11.167166 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-991335-dplhn" podStartSLOduration=2.167151867 podStartE2EDuration="2.167151867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:11.16614952 +0000 UTC m=+2906.720849309" watchObservedRunningTime="2026-10-30 22:02:11.167151867 +0000 UTC m=+2906.721851643" Oct 30 22:02:12.062727 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:12.062689 4560 generic.go:356] "Generic (PLEG): container finished" podID="404224b2-3856-49b0-a6fe-f28335f2580c" containerID="3b252fd81d0867b3d3cb86e692a3b1ea3e7e4615a7d1bd5a4178b443d8411ebe" exitCode=0 Oct 30 22:02:12.063196 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:12.062756 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-991335-dplhn" event={"ID":"404224b2-3856-49b0-a6fe-f28335f2580c","Type":"ContainerDied","Data":"3b252fd81d0867b3d3cb86e692a3b1ea3e7e4615a7d1bd5a4178b443d8411ebe"} Oct 30 22:02:12.547583 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:12.547540 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-a2dced-7f7d56769b-9fh9l" podUID="55bca164-d7e2-4f16-a24d-2aea919e3047" containerName="payload-processing" containerID="cri-o://e250887e902d002a0e7f619d1ad6d1ca7b6bb53a9b4da7b021a8332d57e30444" gracePeriod=25 Oct 30 22:02:13.207054 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:13.207025 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-991335-dplhn" Oct 30 22:02:13.233030 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:13.233000 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/404224b2-3856-49b0-a6fe-f28335f2580c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/404224b2-3856-49b0-a6fe-f28335f2580c-maas-api-xks-ca\") pod \"404224b2-3856-49b0-a6fe-f28335f2580c\" (UID: \"404224b2-3856-49b0-a6fe-f28335f2580c\") " Oct 30 22:02:13.233159 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:13.233077 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/404224b2-3856-49b0-a6fe-f28335f2580c-kube-api-access-pr6pd\" (UniqueName: \"kubernetes.io/projected/404224b2-3856-49b0-a6fe-f28335f2580c-kube-api-access-pr6pd\") pod \"404224b2-3856-49b0-a6fe-f28335f2580c\" (UID: \"404224b2-3856-49b0-a6fe-f28335f2580c\") " Oct 30 22:02:13.233159 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:13.233111 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/404224b2-3856-49b0-a6fe-f28335f2580c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/404224b2-3856-49b0-a6fe-f28335f2580c-maas-api-openshift-ca\") pod \"404224b2-3856-49b0-a6fe-f28335f2580c\" (UID: \"404224b2-3856-49b0-a6fe-f28335f2580c\") " Oct 30 22:02:13.233524 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:13.233496 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/404224b2-3856-49b0-a6fe-f28335f2580c-maas-api-openshift-ca" pod "404224b2-3856-49b0-a6fe-f28335f2580c" (UID: "404224b2-3856-49b0-a6fe-f28335f2580c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:13.235108 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:13.235083 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/404224b2-3856-49b0-a6fe-f28335f2580c-maas-api-xks-ca" pod "404224b2-3856-49b0-a6fe-f28335f2580c" (UID: "404224b2-3856-49b0-a6fe-f28335f2580c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:13.235108 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:13.235099 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/404224b2-3856-49b0-a6fe-f28335f2580c-kube-api-access-pr6pd" pod "404224b2-3856-49b0-a6fe-f28335f2580c" (UID: "404224b2-3856-49b0-a6fe-f28335f2580c"). InnerVolumeSpecName "kube-api-access-pr6pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:13.334718 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:13.334630 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/404224b2-3856-49b0-a6fe-f28335f2580c-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:13.334718 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:13.334670 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr6pd\" (UniqueName: \"kubernetes.io/projected/404224b2-3856-49b0-a6fe-f28335f2580c-kube-api-access-pr6pd\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:13.334718 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:13.334689 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/404224b2-3856-49b0-a6fe-f28335f2580c-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:14.076567 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:14.076532 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-991335-dplhn" event={"ID":"404224b2-3856-49b0-a6fe-f28335f2580c","Type":"ContainerDied","Data":"b0364a5feaea848ebf1952fa3284ed1cb3b2265468f8e18c373ea049e1b69707"} Oct 30 22:02:14.076567 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:14.076553 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-991335-dplhn" Oct 30 22:02:14.076791 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:14.076586 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0364a5feaea848ebf1952fa3284ed1cb3b2265468f8e18c373ea049e1b69707" Oct 30 22:02:14.117951 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:14.117918 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d0bdf9b-gv5rp"] Oct 30 22:02:14.118464 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:14.118446 4560 cpu_manager.go:414] "containerMap: removing container" podUID="404224b2-3856-49b0-a6fe-f28335f2580c" containerName="revoke-keys" Oct 30 22:02:14.118464 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:14.118464 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="404224b2-3856-49b0-a6fe-f28335f2580c" containerName="revoke-keys" Oct 30 22:02:14.118619 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:14.118555 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="404224b2-3856-49b0-a6fe-f28335f2580c" containerName="revoke-keys" Oct 30 22:02:14.131293 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:14.131263 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0bdf9b-gv5rp" Oct 30 22:02:14.166002 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:14.165969 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5f4fd8f-hrps7"] Oct 30 22:02:14.170423 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:14.170399 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f4fd8f-hrps7" Oct 30 22:02:14.180743 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:14.180710 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d0bdf9b-gv5rp"] Oct 30 22:02:14.182648 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:14.182602 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:02:14.185841 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:14.185819 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:02:14.186076 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:14.186042 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:02:14.208918 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:14.208839 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:02:14.247808 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:14.247776 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjf2w\" (UniqueName: \"kubernetes.io/projected/2eecc880-6f0d-4a2c-888e-32803ef8c86b-kube-api-access-sjf2w\") pod \"maas-api-revoke-sub-c5f4fd8f-hrps7\" (UID: \"2eecc880-6f0d-4a2c-888e-32803ef8c86b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f4fd8f-hrps7" Oct 30 22:02:14.247992 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:14.247831 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rnt6\" (UniqueName: \"kubernetes.io/projected/81ad1290-d768-489d-9bcf-2e8f9214b369-kube-api-access-6rnt6\") pod \"maas-api-revoke-sub-7d0bdf9b-gv5rp\" (UID: \"81ad1290-d768-489d-9bcf-2e8f9214b369\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0bdf9b-gv5rp" Oct 30 22:02:14.247992 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:14.247858 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81ad1290-d768-489d-9bcf-2e8f9214b369-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d0bdf9b-gv5rp\" (UID: \"81ad1290-d768-489d-9bcf-2e8f9214b369\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0bdf9b-gv5rp" Oct 30 22:02:14.248089 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:14.248019 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eecc880-6f0d-4a2c-888e-32803ef8c86b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5f4fd8f-hrps7\" (UID: \"2eecc880-6f0d-4a2c-888e-32803ef8c86b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f4fd8f-hrps7" Oct 30 22:02:14.248089 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:14.248069 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81ad1290-d768-489d-9bcf-2e8f9214b369-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d0bdf9b-gv5rp\" (UID: \"81ad1290-d768-489d-9bcf-2e8f9214b369\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0bdf9b-gv5rp" Oct 30 22:02:14.248193 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:14.248178 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eecc880-6f0d-4a2c-888e-32803ef8c86b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5f4fd8f-hrps7\" (UID: \"2eecc880-6f0d-4a2c-888e-32803ef8c86b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f4fd8f-hrps7" Oct 30 22:02:14.276526 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:14.276408 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5f4fd8f-hrps7"] Oct 30 22:02:14.349899 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:14.349812 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eecc880-6f0d-4a2c-888e-32803ef8c86b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5f4fd8f-hrps7\" (UID: \"2eecc880-6f0d-4a2c-888e-32803ef8c86b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f4fd8f-hrps7" Oct 30 22:02:14.350042 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:14.349941 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjf2w\" (UniqueName: \"kubernetes.io/projected/2eecc880-6f0d-4a2c-888e-32803ef8c86b-kube-api-access-sjf2w\") pod \"maas-api-revoke-sub-c5f4fd8f-hrps7\" (UID: \"2eecc880-6f0d-4a2c-888e-32803ef8c86b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f4fd8f-hrps7" Oct 30 22:02:14.350042 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:14.350004 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rnt6\" (UniqueName: \"kubernetes.io/projected/81ad1290-d768-489d-9bcf-2e8f9214b369-kube-api-access-6rnt6\") pod \"maas-api-revoke-sub-7d0bdf9b-gv5rp\" (UID: \"81ad1290-d768-489d-9bcf-2e8f9214b369\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0bdf9b-gv5rp" Oct 30 22:02:14.350160 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:14.350040 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81ad1290-d768-489d-9bcf-2e8f9214b369-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d0bdf9b-gv5rp\" (UID: \"81ad1290-d768-489d-9bcf-2e8f9214b369\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0bdf9b-gv5rp" Oct 30 22:02:14.350160 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:14.350107 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eecc880-6f0d-4a2c-888e-32803ef8c86b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5f4fd8f-hrps7\" (UID: \"2eecc880-6f0d-4a2c-888e-32803ef8c86b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f4fd8f-hrps7" Oct 30 22:02:14.350289 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:14.350166 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81ad1290-d768-489d-9bcf-2e8f9214b369-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d0bdf9b-gv5rp\" (UID: \"81ad1290-d768-489d-9bcf-2e8f9214b369\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0bdf9b-gv5rp" Oct 30 22:02:14.350769 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:14.350746 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81ad1290-d768-489d-9bcf-2e8f9214b369-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d0bdf9b-gv5rp\" (UID: \"81ad1290-d768-489d-9bcf-2e8f9214b369\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0bdf9b-gv5rp" Oct 30 22:02:14.350985 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:14.350957 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eecc880-6f0d-4a2c-888e-32803ef8c86b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5f4fd8f-hrps7\" (UID: \"2eecc880-6f0d-4a2c-888e-32803ef8c86b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f4fd8f-hrps7" Oct 30 22:02:14.352274 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:14.352252 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eecc880-6f0d-4a2c-888e-32803ef8c86b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5f4fd8f-hrps7\" (UID: \"2eecc880-6f0d-4a2c-888e-32803ef8c86b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f4fd8f-hrps7" Oct 30 22:02:14.352717 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:14.352695 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81ad1290-d768-489d-9bcf-2e8f9214b369-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d0bdf9b-gv5rp\" (UID: \"81ad1290-d768-489d-9bcf-2e8f9214b369\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0bdf9b-gv5rp" Oct 30 22:02:14.413357 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:14.413313 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjf2w\" (UniqueName: \"kubernetes.io/projected/2eecc880-6f0d-4a2c-888e-32803ef8c86b-kube-api-access-sjf2w\") pod \"maas-api-revoke-sub-c5f4fd8f-hrps7\" (UID: \"2eecc880-6f0d-4a2c-888e-32803ef8c86b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f4fd8f-hrps7" Oct 30 22:02:14.413635 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:14.413612 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rnt6\" (UniqueName: \"kubernetes.io/projected/81ad1290-d768-489d-9bcf-2e8f9214b369-kube-api-access-6rnt6\") pod \"maas-api-revoke-sub-7d0bdf9b-gv5rp\" (UID: \"81ad1290-d768-489d-9bcf-2e8f9214b369\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0bdf9b-gv5rp" Oct 30 22:02:14.447361 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:14.445307 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0bdf9b-gv5rp" Oct 30 22:02:14.499361 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:14.493576 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f4fd8f-hrps7" Oct 30 22:02:14.925246 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:14.925213 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d0bdf9b-gv5rp"] Oct 30 22:02:14.969949 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:02:14.969921 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81ad1290_d768_489d_9bcf_2e8f9214b369.slice/crio-ee6f1af2de990d566fe0277dd98eb4e1c1355acdb7d26713a832047e57d3cc2b WatchSource:0}: Error finding container ee6f1af2de990d566fe0277dd98eb4e1c1355acdb7d26713a832047e57d3cc2b: Status 404 returned error can't find the container with id ee6f1af2de990d566fe0277dd98eb4e1c1355acdb7d26713a832047e57d3cc2b Oct 30 22:02:14.996304 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:14.996281 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5f4fd8f-hrps7"] Oct 30 22:02:15.084163 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:15.084130 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0bdf9b-gv5rp" event={"ID":"81ad1290-d768-489d-9bcf-2e8f9214b369","Type":"ContainerStarted","Data":"ee6f1af2de990d566fe0277dd98eb4e1c1355acdb7d26713a832047e57d3cc2b"} Oct 30 22:02:15.085659 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:15.085636 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f4fd8f-hrps7" event={"ID":"2eecc880-6f0d-4a2c-888e-32803ef8c86b","Type":"ContainerStarted","Data":"499c9d8cdb7ad998ecec432f78b6c8f7cac5e53ed142ef8f77fc4c7993e1a91f"} Oct 30 22:02:15.829724 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:15.829684 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" podUID="b80b0fba-3830-4a74-a724-aead35443b89" containerName="main" containerID="cri-o://77ebdf1dfd8681c99ec738e440f0b16ceca7abea33d89c730c9cfdedfc551370" gracePeriod=45 Oct 30 22:02:15.845496 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:15.845467 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" podUID="8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3" containerName="main" containerID="cri-o://720a70d747347c95ce3b6c6f00aa7fd196bcde1c7d7664f352102da28ea3bc79" gracePeriod=45 Oct 30 22:02:15.866514 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:15.866483 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" podUID="45a5bac0-28ab-4c78-856e-303f6d4522de" containerName="main" containerID="cri-o://0b9c22a19015ef176dc9de336fdbb6ad4c24de790893208a6d6bd106da2ff3ed" gracePeriod=45 Oct 30 22:02:15.925077 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:15.924719 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" podUID="b80b0fba-3830-4a74-a724-aead35443b89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": dial tcp 10.133.0.80:8000: connect: connection refused" Oct 30 22:02:16.097419 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.097332 4560 generic.go:356] "Generic (PLEG): container finished" podID="2eecc880-6f0d-4a2c-888e-32803ef8c86b" containerID="4d9ff79d527eeabe41a7c6973ed147bec075871a39e6a0e61c0187fdac6949b9" exitCode=0 Oct 30 22:02:16.097631 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.097450 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f4fd8f-hrps7" event={"ID":"2eecc880-6f0d-4a2c-888e-32803ef8c86b","Type":"ContainerDied","Data":"4d9ff79d527eeabe41a7c6973ed147bec075871a39e6a0e61c0187fdac6949b9"} Oct 30 22:02:16.100046 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.100016 4560 generic.go:356] "Generic (PLEG): container finished" podID="b80b0fba-3830-4a74-a724-aead35443b89" containerID="77ebdf1dfd8681c99ec738e440f0b16ceca7abea33d89c730c9cfdedfc551370" exitCode=0 Oct 30 22:02:16.100683 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.100122 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" event={"ID":"b80b0fba-3830-4a74-a724-aead35443b89","Type":"ContainerDied","Data":"77ebdf1dfd8681c99ec738e440f0b16ceca7abea33d89c730c9cfdedfc551370"} Oct 30 22:02:16.102251 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.102229 4560 generic.go:356] "Generic (PLEG): container finished" podID="45a5bac0-28ab-4c78-856e-303f6d4522de" containerID="0b9c22a19015ef176dc9de336fdbb6ad4c24de790893208a6d6bd106da2ff3ed" exitCode=0 Oct 30 22:02:16.102398 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.102288 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" event={"ID":"45a5bac0-28ab-4c78-856e-303f6d4522de","Type":"ContainerDied","Data":"0b9c22a19015ef176dc9de336fdbb6ad4c24de790893208a6d6bd106da2ff3ed"} Oct 30 22:02:16.104599 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.104571 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0bdf9b-gv5rp" event={"ID":"81ad1290-d768-489d-9bcf-2e8f9214b369","Type":"ContainerStarted","Data":"e7e28f7aba6690732cd464b8fd289fd94f65364a45f37e7d5618c5b17a338916"} Oct 30 22:02:16.107612 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.107583 4560 generic.go:356] "Generic (PLEG): container finished" podID="8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3" containerID="720a70d747347c95ce3b6c6f00aa7fd196bcde1c7d7664f352102da28ea3bc79" exitCode=0 Oct 30 22:02:16.107710 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.107697 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" event={"ID":"8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3","Type":"ContainerDied","Data":"720a70d747347c95ce3b6c6f00aa7fd196bcde1c7d7664f352102da28ea3bc79"} Oct 30 22:02:16.207871 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.207841 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" Oct 30 22:02:16.237299 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.237265 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" Oct 30 22:02:16.280290 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.280254 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" Oct 30 22:02:16.311209 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.311177 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b80b0fba-3830-4a74-a724-aead35443b89-tls-certs\" (UniqueName: \"kubernetes.io/secret/b80b0fba-3830-4a74-a724-aead35443b89-tls-certs\") pod \"b80b0fba-3830-4a74-a724-aead35443b89\" (UID: \"b80b0fba-3830-4a74-a724-aead35443b89\") " Oct 30 22:02:16.311381 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.311289 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b80b0fba-3830-4a74-a724-aead35443b89-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b80b0fba-3830-4a74-a724-aead35443b89-tmp-dir\") pod \"b80b0fba-3830-4a74-a724-aead35443b89\" (UID: \"b80b0fba-3830-4a74-a724-aead35443b89\") " Oct 30 22:02:16.311381 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.311352 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b80b0fba-3830-4a74-a724-aead35443b89-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b80b0fba-3830-4a74-a724-aead35443b89-model-cache\") pod \"b80b0fba-3830-4a74-a724-aead35443b89\" (UID: \"b80b0fba-3830-4a74-a724-aead35443b89\") " Oct 30 22:02:16.311599 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.311569 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b80b0fba-3830-4a74-a724-aead35443b89-kube-api-access-6gnlf\" (UniqueName: \"kubernetes.io/projected/b80b0fba-3830-4a74-a724-aead35443b89-kube-api-access-6gnlf\") pod \"b80b0fba-3830-4a74-a724-aead35443b89\" (UID: \"b80b0fba-3830-4a74-a724-aead35443b89\") " Oct 30 22:02:16.311599 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.311589 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b80b0fba-3830-4a74-a724-aead35443b89-model-cache" pod "b80b0fba-3830-4a74-a724-aead35443b89" (UID: "b80b0fba-3830-4a74-a724-aead35443b89"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:16.311721 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.311606 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b80b0fba-3830-4a74-a724-aead35443b89-tmp-dir" pod "b80b0fba-3830-4a74-a724-aead35443b89" (UID: "b80b0fba-3830-4a74-a724-aead35443b89"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:16.311721 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.311616 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-model-cache\") pod \"8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3\" (UID: \"8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3\") " Oct 30 22:02:16.311721 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.311684 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45a5bac0-28ab-4c78-856e-303f6d4522de-tls-certs\" (UniqueName: \"kubernetes.io/secret/45a5bac0-28ab-4c78-856e-303f6d4522de-tls-certs\") pod \"45a5bac0-28ab-4c78-856e-303f6d4522de\" (UID: \"45a5bac0-28ab-4c78-856e-303f6d4522de\") " Oct 30 22:02:16.311950 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.311764 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a5bac0-28ab-4c78-856e-303f6d4522de-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45a5bac0-28ab-4c78-856e-303f6d4522de-model-cache\") pod \"45a5bac0-28ab-4c78-856e-303f6d4522de\" (UID: \"45a5bac0-28ab-4c78-856e-303f6d4522de\") " Oct 30 22:02:16.311950 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.311796 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-model-cache" pod "8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3" (UID: "8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:16.311950 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.311838 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b80b0fba-3830-4a74-a724-aead35443b89-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b80b0fba-3830-4a74-a724-aead35443b89-dshm\") pod \"b80b0fba-3830-4a74-a724-aead35443b89\" (UID: \"b80b0fba-3830-4a74-a724-aead35443b89\") " Oct 30 22:02:16.311950 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.311872 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-home\" (UniqueName: \"kubernetes.io/empty-dir/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-home\") pod \"8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3\" (UID: \"8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3\") " Oct 30 22:02:16.311950 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.311898 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a5bac0-28ab-4c78-856e-303f6d4522de-home\" (UniqueName: \"kubernetes.io/empty-dir/45a5bac0-28ab-4c78-856e-303f6d4522de-home\") pod \"45a5bac0-28ab-4c78-856e-303f6d4522de\" (UID: \"45a5bac0-28ab-4c78-856e-303f6d4522de\") " Oct 30 22:02:16.311950 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.311931 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b80b0fba-3830-4a74-a724-aead35443b89-home\" (UniqueName: \"kubernetes.io/empty-dir/b80b0fba-3830-4a74-a724-aead35443b89-home\") pod \"b80b0fba-3830-4a74-a724-aead35443b89\" (UID: \"b80b0fba-3830-4a74-a724-aead35443b89\") " Oct 30 22:02:16.312278 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.311964 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a5bac0-28ab-4c78-856e-303f6d4522de-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45a5bac0-28ab-4c78-856e-303f6d4522de-tmp-dir\") pod \"45a5bac0-28ab-4c78-856e-303f6d4522de\" (UID: \"45a5bac0-28ab-4c78-856e-303f6d4522de\") " Oct 30 22:02:16.312278 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.311990 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-tls-certs\" (UniqueName: \"kubernetes.io/secret/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-tls-certs\") pod \"8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3\" (UID: \"8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3\") " Oct 30 22:02:16.312278 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.312017 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a5bac0-28ab-4c78-856e-303f6d4522de-model-cache" pod "45a5bac0-28ab-4c78-856e-303f6d4522de" (UID: "45a5bac0-28ab-4c78-856e-303f6d4522de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:16.312278 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.312042 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-dshm\") pod \"8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3\" (UID: \"8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3\") " Oct 30 22:02:16.312278 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.312210 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b80b0fba-3830-4a74-a724-aead35443b89-home" pod "b80b0fba-3830-4a74-a724-aead35443b89" (UID: "b80b0fba-3830-4a74-a724-aead35443b89"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:16.312278 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.312230 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a5bac0-28ab-4c78-856e-303f6d4522de-tmp-dir" pod "45a5bac0-28ab-4c78-856e-303f6d4522de" (UID: "45a5bac0-28ab-4c78-856e-303f6d4522de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:16.312609 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.312431 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-home" pod "8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3" (UID: "8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:16.312609 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.312436 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a5bac0-28ab-4c78-856e-303f6d4522de-home" pod "45a5bac0-28ab-4c78-856e-303f6d4522de" (UID: "45a5bac0-28ab-4c78-856e-303f6d4522de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:16.312723 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.312620 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b80b0fba-3830-4a74-a724-aead35443b89-tmp-dir\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:16.312723 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.312638 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b80b0fba-3830-4a74-a724-aead35443b89-model-cache\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:16.312723 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.312652 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-model-cache\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:16.312723 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.312668 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45a5bac0-28ab-4c78-856e-303f6d4522de-model-cache\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:16.312723 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.312681 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-home\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:16.312723 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.312693 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45a5bac0-28ab-4c78-856e-303f6d4522de-home\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:16.312723 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.312705 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b80b0fba-3830-4a74-a724-aead35443b89-home\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:16.312723 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.312716 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45a5bac0-28ab-4c78-856e-303f6d4522de-tmp-dir\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:16.314068 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.314042 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b80b0fba-3830-4a74-a724-aead35443b89-tls-certs" pod "b80b0fba-3830-4a74-a724-aead35443b89" (UID: "b80b0fba-3830-4a74-a724-aead35443b89"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:16.314424 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.314394 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b80b0fba-3830-4a74-a724-aead35443b89-kube-api-access-6gnlf" pod "b80b0fba-3830-4a74-a724-aead35443b89" (UID: "b80b0fba-3830-4a74-a724-aead35443b89"). InnerVolumeSpecName "kube-api-access-6gnlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:16.314631 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.314611 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45a5bac0-28ab-4c78-856e-303f6d4522de-tls-certs" pod "45a5bac0-28ab-4c78-856e-303f6d4522de" (UID: "45a5bac0-28ab-4c78-856e-303f6d4522de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:16.314710 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.314661 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b80b0fba-3830-4a74-a724-aead35443b89-dshm" pod "b80b0fba-3830-4a74-a724-aead35443b89" (UID: "b80b0fba-3830-4a74-a724-aead35443b89"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:16.314925 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.314912 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-dshm" pod "8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3" (UID: "8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:16.315846 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.315825 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-tls-certs" pod "8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3" (UID: "8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:16.413653 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.413624 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45a5bac0-28ab-4c78-856e-303f6d4522de-kube-api-access-8b7rp\" (UniqueName: \"kubernetes.io/projected/45a5bac0-28ab-4c78-856e-303f6d4522de-kube-api-access-8b7rp\") pod \"45a5bac0-28ab-4c78-856e-303f6d4522de\" (UID: \"45a5bac0-28ab-4c78-856e-303f6d4522de\") " Oct 30 22:02:16.413802 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.413669 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a5bac0-28ab-4c78-856e-303f6d4522de-dshm\" (UniqueName: \"kubernetes.io/empty-dir/45a5bac0-28ab-4c78-856e-303f6d4522de-dshm\") pod \"45a5bac0-28ab-4c78-856e-303f6d4522de\" (UID: \"45a5bac0-28ab-4c78-856e-303f6d4522de\") " Oct 30 22:02:16.413802 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.413788 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-tmp-dir\") pod \"8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3\" (UID: \"8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3\") " Oct 30 22:02:16.413945 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.413855 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-kube-api-access-7f64c\" (UniqueName: \"kubernetes.io/projected/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-kube-api-access-7f64c\") pod \"8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3\" (UID: \"8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3\") " Oct 30 22:02:16.414097 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.414067 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-tmp-dir" pod "8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3" (UID: "8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:16.414247 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.414232 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-tls-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:16.414317 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.414249 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-dshm\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:16.414317 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.414259 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b80b0fba-3830-4a74-a724-aead35443b89-tls-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:16.414317 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.414269 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-tmp-dir\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:16.414317 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.414278 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gnlf\" (UniqueName: \"kubernetes.io/projected/b80b0fba-3830-4a74-a724-aead35443b89-kube-api-access-6gnlf\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:16.414317 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.414286 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45a5bac0-28ab-4c78-856e-303f6d4522de-tls-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:16.414317 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.414298 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b80b0fba-3830-4a74-a724-aead35443b89-dshm\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:16.415644 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.415619 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45a5bac0-28ab-4c78-856e-303f6d4522de-kube-api-access-8b7rp" pod "45a5bac0-28ab-4c78-856e-303f6d4522de" (UID: "45a5bac0-28ab-4c78-856e-303f6d4522de"). InnerVolumeSpecName "kube-api-access-8b7rp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:16.415864 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.415838 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a5bac0-28ab-4c78-856e-303f6d4522de-dshm" pod "45a5bac0-28ab-4c78-856e-303f6d4522de" (UID: "45a5bac0-28ab-4c78-856e-303f6d4522de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:16.415959 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.415883 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-kube-api-access-7f64c" pod "8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3" (UID: "8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3"). InnerVolumeSpecName "kube-api-access-7f64c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:16.515514 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.515485 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f64c\" (UniqueName: \"kubernetes.io/projected/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3-kube-api-access-7f64c\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:16.515514 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.515514 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b7rp\" (UniqueName: \"kubernetes.io/projected/45a5bac0-28ab-4c78-856e-303f6d4522de-kube-api-access-8b7rp\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:16.515728 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.515529 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45a5bac0-28ab-4c78-856e-303f6d4522de-dshm\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:16.613515 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.612684 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71aab5-rt94x"] Oct 30 22:02:16.613515 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:16.612734 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71aab5-rt94x"] Oct 30 22:02:17.116280 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.116251 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" Oct 30 22:02:17.116280 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.116259 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9" event={"ID":"b80b0fba-3830-4a74-a724-aead35443b89","Type":"ContainerDied","Data":"2b72ee34d5d7cdcb1ff87a8af0e694744e65051a3c066d4463d6e0a527310b8b"} Oct 30 22:02:17.116858 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.116300 4560 scope.go:122] "RemoveContainer" containerID="77ebdf1dfd8681c99ec738e440f0b16ceca7abea33d89c730c9cfdedfc551370" Oct 30 22:02:17.117905 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.117884 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" Oct 30 22:02:17.118030 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.117886 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q" event={"ID":"45a5bac0-28ab-4c78-856e-303f6d4522de","Type":"ContainerDied","Data":"9b550b68cb3708034bbcb41d70ed45de4aaf4116ed0081909ebd3c8e17f16aa8"} Oct 30 22:02:17.119753 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.119730 4560 generic.go:356] "Generic (PLEG): container finished" podID="81ad1290-d768-489d-9bcf-2e8f9214b369" containerID="e7e28f7aba6690732cd464b8fd289fd94f65364a45f37e7d5618c5b17a338916" exitCode=0 Oct 30 22:02:17.119912 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.119848 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0bdf9b-gv5rp" event={"ID":"81ad1290-d768-489d-9bcf-2e8f9214b369","Type":"ContainerDied","Data":"e7e28f7aba6690732cd464b8fd289fd94f65364a45f37e7d5618c5b17a338916"} Oct 30 22:02:17.122232 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.122216 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" Oct 30 22:02:17.122331 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.122262 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" event={"ID":"8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3","Type":"ContainerDied","Data":"e26ec883712965a4d231cfb4fbcf41e5baaef25bec6b604309ec4aad3de0b899"} Oct 30 22:02:17.126969 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.126948 4560 scope.go:122] "RemoveContainer" containerID="0b9c22a19015ef176dc9de336fdbb6ad4c24de790893208a6d6bd106da2ff3ed" Oct 30 22:02:17.135958 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.135938 4560 scope.go:122] "RemoveContainer" containerID="720a70d747347c95ce3b6c6f00aa7fd196bcde1c7d7664f352102da28ea3bc79" Oct 30 22:02:17.210974 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.210785 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p" podUID="8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:02:17.279229 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.279204 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f4fd8f-hrps7" Oct 30 22:02:17.289577 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.289548 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0bdf9b-gv5rp" Oct 30 22:02:17.290895 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.290872 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p"] Oct 30 22:02:17.303780 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.300540 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-a2dced/premium-w6-a2dced-kserve-86778c4b8f-jbn5p"] Oct 30 22:02:17.334246 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.334225 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81ad1290-d768-489d-9bcf-2e8f9214b369-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81ad1290-d768-489d-9bcf-2e8f9214b369-maas-api-openshift-ca\") pod \"81ad1290-d768-489d-9bcf-2e8f9214b369\" (UID: \"81ad1290-d768-489d-9bcf-2e8f9214b369\") " Oct 30 22:02:17.334417 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.334399 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2eecc880-6f0d-4a2c-888e-32803ef8c86b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eecc880-6f0d-4a2c-888e-32803ef8c86b-maas-api-xks-ca\") pod \"2eecc880-6f0d-4a2c-888e-32803ef8c86b\" (UID: \"2eecc880-6f0d-4a2c-888e-32803ef8c86b\") " Oct 30 22:02:17.334472 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.334440 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81ad1290-d768-489d-9bcf-2e8f9214b369-kube-api-access-6rnt6\" (UniqueName: \"kubernetes.io/projected/81ad1290-d768-489d-9bcf-2e8f9214b369-kube-api-access-6rnt6\") pod \"81ad1290-d768-489d-9bcf-2e8f9214b369\" (UID: \"81ad1290-d768-489d-9bcf-2e8f9214b369\") " Oct 30 22:02:17.334518 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.334489 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eecc880-6f0d-4a2c-888e-32803ef8c86b-kube-api-access-sjf2w\" (UniqueName: \"kubernetes.io/projected/2eecc880-6f0d-4a2c-888e-32803ef8c86b-kube-api-access-sjf2w\") pod \"2eecc880-6f0d-4a2c-888e-32803ef8c86b\" (UID: \"2eecc880-6f0d-4a2c-888e-32803ef8c86b\") " Oct 30 22:02:17.334572 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.334514 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81ad1290-d768-489d-9bcf-2e8f9214b369-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81ad1290-d768-489d-9bcf-2e8f9214b369-maas-api-xks-ca\") pod \"81ad1290-d768-489d-9bcf-2e8f9214b369\" (UID: \"81ad1290-d768-489d-9bcf-2e8f9214b369\") " Oct 30 22:02:17.334572 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.334550 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eecc880-6f0d-4a2c-888e-32803ef8c86b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eecc880-6f0d-4a2c-888e-32803ef8c86b-maas-api-openshift-ca\") pod \"2eecc880-6f0d-4a2c-888e-32803ef8c86b\" (UID: \"2eecc880-6f0d-4a2c-888e-32803ef8c86b\") " Oct 30 22:02:17.334697 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.334650 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81ad1290-d768-489d-9bcf-2e8f9214b369-maas-api-openshift-ca" pod "81ad1290-d768-489d-9bcf-2e8f9214b369" (UID: "81ad1290-d768-489d-9bcf-2e8f9214b369"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:17.335055 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.335034 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81ad1290-d768-489d-9bcf-2e8f9214b369-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:17.335452 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.335423 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eecc880-6f0d-4a2c-888e-32803ef8c86b-maas-api-openshift-ca" pod "2eecc880-6f0d-4a2c-888e-32803ef8c86b" (UID: "2eecc880-6f0d-4a2c-888e-32803ef8c86b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:17.336789 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.336763 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2eecc880-6f0d-4a2c-888e-32803ef8c86b-maas-api-xks-ca" pod "2eecc880-6f0d-4a2c-888e-32803ef8c86b" (UID: "2eecc880-6f0d-4a2c-888e-32803ef8c86b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:17.336881 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.336809 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81ad1290-d768-489d-9bcf-2e8f9214b369-kube-api-access-6rnt6" pod "81ad1290-d768-489d-9bcf-2e8f9214b369" (UID: "81ad1290-d768-489d-9bcf-2e8f9214b369"). InnerVolumeSpecName "kube-api-access-6rnt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:17.336881 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.336837 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81ad1290-d768-489d-9bcf-2e8f9214b369-maas-api-xks-ca" pod "81ad1290-d768-489d-9bcf-2e8f9214b369" (UID: "81ad1290-d768-489d-9bcf-2e8f9214b369"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:17.337123 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.337102 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eecc880-6f0d-4a2c-888e-32803ef8c86b-kube-api-access-sjf2w" pod "2eecc880-6f0d-4a2c-888e-32803ef8c86b" (UID: "2eecc880-6f0d-4a2c-888e-32803ef8c86b"). InnerVolumeSpecName "kube-api-access-sjf2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:17.414249 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.414173 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9"] Oct 30 22:02:17.434675 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.434646 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:02:17.435642 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.435625 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjf2w\" (UniqueName: \"kubernetes.io/projected/2eecc880-6f0d-4a2c-888e-32803ef8c86b-kube-api-access-sjf2w\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:17.435687 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.435646 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81ad1290-d768-489d-9bcf-2e8f9214b369-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:17.435687 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.435658 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eecc880-6f0d-4a2c-888e-32803ef8c86b-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:17.435687 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.435677 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eecc880-6f0d-4a2c-888e-32803ef8c86b-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:17.435795 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.435687 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rnt6\" (UniqueName: \"kubernetes.io/projected/81ad1290-d768-489d-9bcf-2e8f9214b369-kube-api-access-6rnt6\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:17.439926 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.439903 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-a2dced/distinct-2-w6-a2dced-kserve-897cb76c-lz5w9"] Oct 30 22:02:17.565966 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.565933 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q"] Oct 30 22:02:17.575398 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.575368 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-a2dced/simulator-w6-a2dced-kserve-5f7fc45cfb-4kp6q"] Oct 30 22:02:17.824159 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.824126 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19dd4f-tg7k6"] Oct 30 22:02:17.926787 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:17.926746 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19dd4f-tg7k6"] Oct 30 22:02:18.129096 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:18.129010 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f4fd8f-hrps7" Oct 30 22:02:18.129096 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:18.129005 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f4fd8f-hrps7" event={"ID":"2eecc880-6f0d-4a2c-888e-32803ef8c86b","Type":"ContainerDied","Data":"499c9d8cdb7ad998ecec432f78b6c8f7cac5e53ed142ef8f77fc4c7993e1a91f"} Oct 30 22:02:18.129096 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:18.129064 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="499c9d8cdb7ad998ecec432f78b6c8f7cac5e53ed142ef8f77fc4c7993e1a91f" Oct 30 22:02:18.132401 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:18.132373 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0bdf9b-gv5rp" Oct 30 22:02:18.132401 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:18.132380 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0bdf9b-gv5rp" event={"ID":"81ad1290-d768-489d-9bcf-2e8f9214b369","Type":"ContainerDied","Data":"ee6f1af2de990d566fe0277dd98eb4e1c1355acdb7d26713a832047e57d3cc2b"} Oct 30 22:02:18.132617 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:18.132428 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee6f1af2de990d566fe0277dd98eb4e1c1355acdb7d26713a832047e57d3cc2b" Oct 30 22:02:18.574916 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:18.574883 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="363ad13f-c717-4b4e-a4fc-f114c21b5d8f" path="/var/lib/kubelet/pods/363ad13f-c717-4b4e-a4fc-f114c21b5d8f/volumes" Oct 30 22:02:18.575395 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:18.575376 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45a5bac0-28ab-4c78-856e-303f6d4522de" path="/var/lib/kubelet/pods/45a5bac0-28ab-4c78-856e-303f6d4522de/volumes" Oct 30 22:02:18.575823 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:18.575808 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3" path="/var/lib/kubelet/pods/8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3/volumes" Oct 30 22:02:18.576457 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:18.576436 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b80b0fba-3830-4a74-a724-aead35443b89" path="/var/lib/kubelet/pods/b80b0fba-3830-4a74-a724-aead35443b89/volumes" Oct 30 22:02:18.576868 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:18.576854 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c965fda1-f5b4-402a-99c7-f9ec481edabd" path="/var/lib/kubelet/pods/c965fda1-f5b4-402a-99c7-f9ec481edabd/volumes" Oct 30 22:02:19.684024 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:02:19.683989 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:02:19.842131 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:19.842100 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf"] Oct 30 22:02:19.842330 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:19.842307 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" podUID="156f466e-16e0-441c-9ce5-796acd8551b2" containerName="istio-proxy" containerID="cri-o://1ba3e97851e342b02fe94b46a8d3de3994e2840360a6cb9de76b226ec01f4a27" gracePeriod=30 Oct 30 22:02:19.844385 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:19.844356 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p"] Oct 30 22:02:19.844562 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:19.844541 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" podUID="f87e7e5a-9553-4dcb-9efa-d6557a171331" containerName="istio-proxy" containerID="cri-o://5f1b662cc15575130db940c549390e26b013715fd041c3df354f1a335a1853ab" gracePeriod=30 Oct 30 22:02:19.862664 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:19.862640 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g"] Oct 30 22:02:19.862826 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:19.862808 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" podUID="1e67aa41-ff98-4eab-a76c-020b1de1ff8e" containerName="istio-proxy" containerID="cri-o://587c91fe2d9dbdea9b0b8ac06160d64e0ee973448aa444a51e8880ab1a59d8e0" gracePeriod=30 Oct 30 22:02:22.179966 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:22.179928 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-a2dced-7f7d56769b-9fh9l" podUID="55bca164-d7e2-4f16-a24d-2aea919e3047" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:02:25.153825 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.153797 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" Oct 30 22:02:25.176624 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.176596 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" Oct 30 22:02:25.178750 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.178721 4560 generic.go:356] "Generic (PLEG): container finished" podID="156f466e-16e0-441c-9ce5-796acd8551b2" containerID="1ba3e97851e342b02fe94b46a8d3de3994e2840360a6cb9de76b226ec01f4a27" exitCode=0 Oct 30 22:02:25.178920 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.178806 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" event={"ID":"156f466e-16e0-441c-9ce5-796acd8551b2","Type":"ContainerDied","Data":"1ba3e97851e342b02fe94b46a8d3de3994e2840360a6cb9de76b226ec01f4a27"} Oct 30 22:02:25.178920 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.178874 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" event={"ID":"156f466e-16e0-441c-9ce5-796acd8551b2","Type":"ContainerDied","Data":"b90ead70ad1c49325a286dcf50c0fcbbb31cd727417372ea820beed62032662c"} Oct 30 22:02:25.178920 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.178896 4560 scope.go:122] "RemoveContainer" containerID="1ba3e97851e342b02fe94b46a8d3de3994e2840360a6cb9de76b226ec01f4a27" Oct 30 22:02:25.179094 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.179010 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf" Oct 30 22:02:25.180776 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.180754 4560 generic.go:356] "Generic (PLEG): container finished" podID="f87e7e5a-9553-4dcb-9efa-d6557a171331" containerID="5f1b662cc15575130db940c549390e26b013715fd041c3df354f1a335a1853ab" exitCode=0 Oct 30 22:02:25.180903 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.180858 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" Oct 30 22:02:25.180903 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.180881 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" event={"ID":"f87e7e5a-9553-4dcb-9efa-d6557a171331","Type":"ContainerDied","Data":"5f1b662cc15575130db940c549390e26b013715fd041c3df354f1a335a1853ab"} Oct 30 22:02:25.181012 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.180929 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p" event={"ID":"f87e7e5a-9553-4dcb-9efa-d6557a171331","Type":"ContainerDied","Data":"f38d71cac9388679d2afb166df1dee04b9d9034f5d4479a4999e572018420ece"} Oct 30 22:02:25.184568 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.184543 4560 generic.go:356] "Generic (PLEG): container finished" podID="1e67aa41-ff98-4eab-a76c-020b1de1ff8e" containerID="587c91fe2d9dbdea9b0b8ac06160d64e0ee973448aa444a51e8880ab1a59d8e0" exitCode=0 Oct 30 22:02:25.184697 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.184594 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" event={"ID":"1e67aa41-ff98-4eab-a76c-020b1de1ff8e","Type":"ContainerDied","Data":"587c91fe2d9dbdea9b0b8ac06160d64e0ee973448aa444a51e8880ab1a59d8e0"} Oct 30 22:02:25.193977 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.192730 4560 scope.go:122] "RemoveContainer" containerID="1ba3e97851e342b02fe94b46a8d3de3994e2840360a6cb9de76b226ec01f4a27" Oct 30 22:02:25.193977 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:02:25.193082 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ba3e97851e342b02fe94b46a8d3de3994e2840360a6cb9de76b226ec01f4a27\": container with ID starting with 1ba3e97851e342b02fe94b46a8d3de3994e2840360a6cb9de76b226ec01f4a27 not found: ID does not exist" containerID="1ba3e97851e342b02fe94b46a8d3de3994e2840360a6cb9de76b226ec01f4a27" Oct 30 22:02:25.193977 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.193119 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ba3e97851e342b02fe94b46a8d3de3994e2840360a6cb9de76b226ec01f4a27"} err="failed to get container status \"1ba3e97851e342b02fe94b46a8d3de3994e2840360a6cb9de76b226ec01f4a27\": rpc error: code = NotFound desc = could not find container \"1ba3e97851e342b02fe94b46a8d3de3994e2840360a6cb9de76b226ec01f4a27\": container with ID starting with 1ba3e97851e342b02fe94b46a8d3de3994e2840360a6cb9de76b226ec01f4a27 not found: ID does not exist" Oct 30 22:02:25.193977 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.193141 4560 scope.go:122] "RemoveContainer" containerID="5f1b662cc15575130db940c549390e26b013715fd041c3df354f1a335a1853ab" Oct 30 22:02:25.223879 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.223834 4560 scope.go:122] "RemoveContainer" containerID="5f1b662cc15575130db940c549390e26b013715fd041c3df354f1a335a1853ab" Oct 30 22:02:25.224353 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:02:25.224301 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f1b662cc15575130db940c549390e26b013715fd041c3df354f1a335a1853ab\": container with ID starting with 5f1b662cc15575130db940c549390e26b013715fd041c3df354f1a335a1853ab not found: ID does not exist" containerID="5f1b662cc15575130db940c549390e26b013715fd041c3df354f1a335a1853ab" Oct 30 22:02:25.224477 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.224445 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f1b662cc15575130db940c549390e26b013715fd041c3df354f1a335a1853ab"} err="failed to get container status \"5f1b662cc15575130db940c549390e26b013715fd041c3df354f1a335a1853ab\": rpc error: code = NotFound desc = could not find container \"5f1b662cc15575130db940c549390e26b013715fd041c3df354f1a335a1853ab\": container with ID starting with 5f1b662cc15575130db940c549390e26b013715fd041c3df354f1a335a1853ab not found: ID does not exist" Oct 30 22:02:25.240517 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.240461 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" Oct 30 22:02:25.286022 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.285990 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f87e7e5a-9553-4dcb-9efa-d6557a171331-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f87e7e5a-9553-4dcb-9efa-d6557a171331-istiod-ca-cert\") pod \"f87e7e5a-9553-4dcb-9efa-d6557a171331\" (UID: \"f87e7e5a-9553-4dcb-9efa-d6557a171331\") " Oct 30 22:02:25.286226 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.286032 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f87e7e5a-9553-4dcb-9efa-d6557a171331-istio-token\" (UniqueName: \"kubernetes.io/projected/f87e7e5a-9553-4dcb-9efa-d6557a171331-istio-token\") pod \"f87e7e5a-9553-4dcb-9efa-d6557a171331\" (UID: \"f87e7e5a-9553-4dcb-9efa-d6557a171331\") " Oct 30 22:02:25.286226 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.286058 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f87e7e5a-9553-4dcb-9efa-d6557a171331-kube-api-access-xnjx2\" (UniqueName: \"kubernetes.io/projected/f87e7e5a-9553-4dcb-9efa-d6557a171331-kube-api-access-xnjx2\") pod \"f87e7e5a-9553-4dcb-9efa-d6557a171331\" (UID: \"f87e7e5a-9553-4dcb-9efa-d6557a171331\") " Oct 30 22:02:25.286226 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.286085 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-istio-envoy\") pod \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\" (UID: \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\") " Oct 30 22:02:25.286226 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.286107 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-workload-certs\") pod \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\" (UID: \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\") " Oct 30 22:02:25.286226 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.286123 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f87e7e5a-9553-4dcb-9efa-d6557a171331-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f87e7e5a-9553-4dcb-9efa-d6557a171331-workload-certs\") pod \"f87e7e5a-9553-4dcb-9efa-d6557a171331\" (UID: \"f87e7e5a-9553-4dcb-9efa-d6557a171331\") " Oct 30 22:02:25.286226 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.286175 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f87e7e5a-9553-4dcb-9efa-d6557a171331-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f87e7e5a-9553-4dcb-9efa-d6557a171331-credential-socket\") pod \"f87e7e5a-9553-4dcb-9efa-d6557a171331\" (UID: \"f87e7e5a-9553-4dcb-9efa-d6557a171331\") " Oct 30 22:02:25.286226 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.286200 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f87e7e5a-9553-4dcb-9efa-d6557a171331-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f87e7e5a-9553-4dcb-9efa-d6557a171331-istio-data\") pod \"f87e7e5a-9553-4dcb-9efa-d6557a171331\" (UID: \"f87e7e5a-9553-4dcb-9efa-d6557a171331\") " Oct 30 22:02:25.286659 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.286256 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f87e7e5a-9553-4dcb-9efa-d6557a171331-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f87e7e5a-9553-4dcb-9efa-d6557a171331-istio-envoy\") pod \"f87e7e5a-9553-4dcb-9efa-d6557a171331\" (UID: \"f87e7e5a-9553-4dcb-9efa-d6557a171331\") " Oct 30 22:02:25.286659 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.286282 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-istio-data\") pod \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\" (UID: \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\") " Oct 30 22:02:25.286659 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.286328 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/156f466e-16e0-441c-9ce5-796acd8551b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/156f466e-16e0-441c-9ce5-796acd8551b2-istiod-ca-cert\") pod \"156f466e-16e0-441c-9ce5-796acd8551b2\" (UID: \"156f466e-16e0-441c-9ce5-796acd8551b2\") " Oct 30 22:02:25.286659 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.286394 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-istio-token\" (UniqueName: \"kubernetes.io/projected/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-istio-token\") pod \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\" (UID: \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\") " Oct 30 22:02:25.286659 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.286420 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-credential-socket\") pod \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\" (UID: \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\") " Oct 30 22:02:25.286659 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.286464 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f87e7e5a-9553-4dcb-9efa-d6557a171331-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f87e7e5a-9553-4dcb-9efa-d6557a171331-istio-podinfo\") pod \"f87e7e5a-9553-4dcb-9efa-d6557a171331\" (UID: \"f87e7e5a-9553-4dcb-9efa-d6557a171331\") " Oct 30 22:02:25.286659 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.286531 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/156f466e-16e0-441c-9ce5-796acd8551b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/156f466e-16e0-441c-9ce5-796acd8551b2-istio-data\") pod \"156f466e-16e0-441c-9ce5-796acd8551b2\" (UID: \"156f466e-16e0-441c-9ce5-796acd8551b2\") " Oct 30 22:02:25.286659 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.286456 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f87e7e5a-9553-4dcb-9efa-d6557a171331-istiod-ca-cert" pod "f87e7e5a-9553-4dcb-9efa-d6557a171331" (UID: "f87e7e5a-9553-4dcb-9efa-d6557a171331"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:25.286659 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.286556 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f87e7e5a-9553-4dcb-9efa-d6557a171331-workload-certs" pod "f87e7e5a-9553-4dcb-9efa-d6557a171331" (UID: "f87e7e5a-9553-4dcb-9efa-d6557a171331"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:25.286659 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.286577 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-istio-podinfo\") pod \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\" (UID: \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\") " Oct 30 22:02:25.287167 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.286665 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-workload-certs" pod "1e67aa41-ff98-4eab-a76c-020b1de1ff8e" (UID: "1e67aa41-ff98-4eab-a76c-020b1de1ff8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:25.287167 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.286689 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-workload-socket\") pod \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\" (UID: \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\") " Oct 30 22:02:25.287167 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.286725 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-kube-api-access-4bc8f\" (UniqueName: \"kubernetes.io/projected/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-kube-api-access-4bc8f\") pod \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\" (UID: \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\") " Oct 30 22:02:25.287167 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.286782 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/156f466e-16e0-441c-9ce5-796acd8551b2-istio-token\" (UniqueName: \"kubernetes.io/projected/156f466e-16e0-441c-9ce5-796acd8551b2-istio-token\") pod \"156f466e-16e0-441c-9ce5-796acd8551b2\" (UID: \"156f466e-16e0-441c-9ce5-796acd8551b2\") " Oct 30 22:02:25.287167 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.286796 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f87e7e5a-9553-4dcb-9efa-d6557a171331-credential-socket" pod "f87e7e5a-9553-4dcb-9efa-d6557a171331" (UID: "f87e7e5a-9553-4dcb-9efa-d6557a171331"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:25.287167 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.286850 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f87e7e5a-9553-4dcb-9efa-d6557a171331-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f87e7e5a-9553-4dcb-9efa-d6557a171331-workload-socket\") pod \"f87e7e5a-9553-4dcb-9efa-d6557a171331\" (UID: \"f87e7e5a-9553-4dcb-9efa-d6557a171331\") " Oct 30 22:02:25.287167 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.286890 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/156f466e-16e0-441c-9ce5-796acd8551b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/156f466e-16e0-441c-9ce5-796acd8551b2-credential-socket\") pod \"156f466e-16e0-441c-9ce5-796acd8551b2\" (UID: \"156f466e-16e0-441c-9ce5-796acd8551b2\") " Oct 30 22:02:25.287167 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.286916 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f87e7e5a-9553-4dcb-9efa-d6557a171331-istio-data" pod "f87e7e5a-9553-4dcb-9efa-d6557a171331" (UID: "f87e7e5a-9553-4dcb-9efa-d6557a171331"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:25.287167 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.286955 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/156f466e-16e0-441c-9ce5-796acd8551b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/156f466e-16e0-441c-9ce5-796acd8551b2-istio-envoy\") pod \"156f466e-16e0-441c-9ce5-796acd8551b2\" (UID: \"156f466e-16e0-441c-9ce5-796acd8551b2\") " Oct 30 22:02:25.287167 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.286994 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/156f466e-16e0-441c-9ce5-796acd8551b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/156f466e-16e0-441c-9ce5-796acd8551b2-workload-certs\") pod \"156f466e-16e0-441c-9ce5-796acd8551b2\" (UID: \"156f466e-16e0-441c-9ce5-796acd8551b2\") " Oct 30 22:02:25.287167 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.287035 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/156f466e-16e0-441c-9ce5-796acd8551b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/156f466e-16e0-441c-9ce5-796acd8551b2-istio-podinfo\") pod \"156f466e-16e0-441c-9ce5-796acd8551b2\" (UID: \"156f466e-16e0-441c-9ce5-796acd8551b2\") " Oct 30 22:02:25.287167 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.287068 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/156f466e-16e0-441c-9ce5-796acd8551b2-kube-api-access-sf4mg\" (UniqueName: \"kubernetes.io/projected/156f466e-16e0-441c-9ce5-796acd8551b2-kube-api-access-sf4mg\") pod \"156f466e-16e0-441c-9ce5-796acd8551b2\" (UID: \"156f466e-16e0-441c-9ce5-796acd8551b2\") " Oct 30 22:02:25.287167 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.287104 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/156f466e-16e0-441c-9ce5-796acd8551b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/156f466e-16e0-441c-9ce5-796acd8551b2-workload-socket\") pod \"156f466e-16e0-441c-9ce5-796acd8551b2\" (UID: \"156f466e-16e0-441c-9ce5-796acd8551b2\") " Oct 30 22:02:25.287167 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.287146 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-istiod-ca-cert\") pod \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\" (UID: \"1e67aa41-ff98-4eab-a76c-020b1de1ff8e\") " Oct 30 22:02:25.287853 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.287637 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-workload-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.287853 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.287657 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f87e7e5a-9553-4dcb-9efa-d6557a171331-workload-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.287853 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.287673 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f87e7e5a-9553-4dcb-9efa-d6557a171331-credential-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.287853 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.287687 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f87e7e5a-9553-4dcb-9efa-d6557a171331-istio-data\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.287853 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.287701 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f87e7e5a-9553-4dcb-9efa-d6557a171331-istiod-ca-cert\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.288789 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.286994 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-workload-socket" pod "1e67aa41-ff98-4eab-a76c-020b1de1ff8e" (UID: "1e67aa41-ff98-4eab-a76c-020b1de1ff8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:25.288789 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.288021 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-istiod-ca-cert" pod "1e67aa41-ff98-4eab-a76c-020b1de1ff8e" (UID: "1e67aa41-ff98-4eab-a76c-020b1de1ff8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:25.288978 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.288203 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f87e7e5a-9553-4dcb-9efa-d6557a171331-workload-socket" pod "f87e7e5a-9553-4dcb-9efa-d6557a171331" (UID: "f87e7e5a-9553-4dcb-9efa-d6557a171331"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:25.288978 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.288491 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/156f466e-16e0-441c-9ce5-796acd8551b2-credential-socket" pod "156f466e-16e0-441c-9ce5-796acd8551b2" (UID: "156f466e-16e0-441c-9ce5-796acd8551b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:25.288978 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.288757 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-credential-socket" pod "1e67aa41-ff98-4eab-a76c-020b1de1ff8e" (UID: "1e67aa41-ff98-4eab-a76c-020b1de1ff8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:25.289151 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.289002 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/156f466e-16e0-441c-9ce5-796acd8551b2-istio-data" pod "156f466e-16e0-441c-9ce5-796acd8551b2" (UID: "156f466e-16e0-441c-9ce5-796acd8551b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:25.289298 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.289269 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/156f466e-16e0-441c-9ce5-796acd8551b2-istiod-ca-cert" pod "156f466e-16e0-441c-9ce5-796acd8551b2" (UID: "156f466e-16e0-441c-9ce5-796acd8551b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:25.289713 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.289407 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f87e7e5a-9553-4dcb-9efa-d6557a171331-istio-token" pod "f87e7e5a-9553-4dcb-9efa-d6557a171331" (UID: "f87e7e5a-9553-4dcb-9efa-d6557a171331"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:25.289713 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.289457 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-istio-data" pod "1e67aa41-ff98-4eab-a76c-020b1de1ff8e" (UID: "1e67aa41-ff98-4eab-a76c-020b1de1ff8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:25.290063 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.290031 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/156f466e-16e0-441c-9ce5-796acd8551b2-workload-certs" pod "156f466e-16e0-441c-9ce5-796acd8551b2" (UID: "156f466e-16e0-441c-9ce5-796acd8551b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:25.290152 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.290121 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-istio-envoy" pod "1e67aa41-ff98-4eab-a76c-020b1de1ff8e" (UID: "1e67aa41-ff98-4eab-a76c-020b1de1ff8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:25.290216 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.290170 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f87e7e5a-9553-4dcb-9efa-d6557a171331-istio-podinfo" pod "f87e7e5a-9553-4dcb-9efa-d6557a171331" (UID: "f87e7e5a-9553-4dcb-9efa-d6557a171331"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:02:25.290274 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.290221 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-istio-podinfo" pod "1e67aa41-ff98-4eab-a76c-020b1de1ff8e" (UID: "1e67aa41-ff98-4eab-a76c-020b1de1ff8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:02:25.290473 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.290441 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f87e7e5a-9553-4dcb-9efa-d6557a171331-kube-api-access-xnjx2" pod "f87e7e5a-9553-4dcb-9efa-d6557a171331" (UID: "f87e7e5a-9553-4dcb-9efa-d6557a171331"). InnerVolumeSpecName "kube-api-access-xnjx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:25.290557 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.290468 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/156f466e-16e0-441c-9ce5-796acd8551b2-workload-socket" pod "156f466e-16e0-441c-9ce5-796acd8551b2" (UID: "156f466e-16e0-441c-9ce5-796acd8551b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:25.290557 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.290531 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-istio-token" pod "1e67aa41-ff98-4eab-a76c-020b1de1ff8e" (UID: "1e67aa41-ff98-4eab-a76c-020b1de1ff8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:25.291334 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.291311 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-kube-api-access-4bc8f" pod "1e67aa41-ff98-4eab-a76c-020b1de1ff8e" (UID: "1e67aa41-ff98-4eab-a76c-020b1de1ff8e"). InnerVolumeSpecName "kube-api-access-4bc8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:25.291682 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.291651 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/156f466e-16e0-441c-9ce5-796acd8551b2-istio-token" pod "156f466e-16e0-441c-9ce5-796acd8551b2" (UID: "156f466e-16e0-441c-9ce5-796acd8551b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:25.291754 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.291738 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/156f466e-16e0-441c-9ce5-796acd8551b2-kube-api-access-sf4mg" pod "156f466e-16e0-441c-9ce5-796acd8551b2" (UID: "156f466e-16e0-441c-9ce5-796acd8551b2"). InnerVolumeSpecName "kube-api-access-sf4mg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:25.291825 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.291803 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/156f466e-16e0-441c-9ce5-796acd8551b2-istio-envoy" pod "156f466e-16e0-441c-9ce5-796acd8551b2" (UID: "156f466e-16e0-441c-9ce5-796acd8551b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:25.292322 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.292295 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f87e7e5a-9553-4dcb-9efa-d6557a171331-istio-envoy" pod "f87e7e5a-9553-4dcb-9efa-d6557a171331" (UID: "f87e7e5a-9553-4dcb-9efa-d6557a171331"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:25.292951 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.292935 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/156f466e-16e0-441c-9ce5-796acd8551b2-istio-podinfo" pod "156f466e-16e0-441c-9ce5-796acd8551b2" (UID: "156f466e-16e0-441c-9ce5-796acd8551b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:02:25.405388 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.391141 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-istiod-ca-cert\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.405388 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.391181 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f87e7e5a-9553-4dcb-9efa-d6557a171331-istio-token\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.405388 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.391198 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnjx2\" (UniqueName: \"kubernetes.io/projected/f87e7e5a-9553-4dcb-9efa-d6557a171331-kube-api-access-xnjx2\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.405388 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.391215 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-istio-envoy\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.405388 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.391231 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f87e7e5a-9553-4dcb-9efa-d6557a171331-istio-envoy\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.405388 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.391244 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-istio-data\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.405388 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.391258 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/156f466e-16e0-441c-9ce5-796acd8551b2-istiod-ca-cert\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.405388 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.391273 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-istio-token\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.405388 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.391287 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-credential-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.405388 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.391301 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f87e7e5a-9553-4dcb-9efa-d6557a171331-istio-podinfo\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.405388 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.391314 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/156f466e-16e0-441c-9ce5-796acd8551b2-istio-data\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.405388 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.391326 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-istio-podinfo\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.405388 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.391355 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-workload-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.405388 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.391370 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bc8f\" (UniqueName: \"kubernetes.io/projected/1e67aa41-ff98-4eab-a76c-020b1de1ff8e-kube-api-access-4bc8f\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.405388 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.391384 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/156f466e-16e0-441c-9ce5-796acd8551b2-istio-token\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.405388 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.391398 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f87e7e5a-9553-4dcb-9efa-d6557a171331-workload-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.405388 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.391410 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/156f466e-16e0-441c-9ce5-796acd8551b2-credential-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.405388 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.391424 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/156f466e-16e0-441c-9ce5-796acd8551b2-istio-envoy\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.405388 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.391436 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/156f466e-16e0-441c-9ce5-796acd8551b2-workload-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.405388 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.391448 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/156f466e-16e0-441c-9ce5-796acd8551b2-istio-podinfo\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.405388 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.391467 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf4mg\" (UniqueName: \"kubernetes.io/projected/156f466e-16e0-441c-9ce5-796acd8551b2-kube-api-access-sf4mg\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.405388 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.391480 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/156f466e-16e0-441c-9ce5-796acd8551b2-workload-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.734134 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.734096 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p"] Oct 30 22:02:25.739536 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.739505 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-rzz7p"] Oct 30 22:02:25.851121 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.851087 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf"] Oct 30 22:02:25.861829 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:25.861802 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-2w9jf"] Oct 30 22:02:26.194429 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:26.193874 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" Oct 30 22:02:26.194429 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:26.193886 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g" event={"ID":"1e67aa41-ff98-4eab-a76c-020b1de1ff8e","Type":"ContainerDied","Data":"ee07d53d88fa7b37cd2fc3203cb1751dbb11cc60ca5579cb23a255ddcc642fba"} Oct 30 22:02:26.194429 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:26.193979 4560 scope.go:122] "RemoveContainer" containerID="587c91fe2d9dbdea9b0b8ac06160d64e0ee973448aa444a51e8880ab1a59d8e0" Oct 30 22:02:26.370869 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:26.370820 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g"] Oct 30 22:02:26.373206 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:26.373176 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-bfdf30b9-openshift-default-8486754c49-6nd4g"] Oct 30 22:02:26.577178 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:26.577146 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="156f466e-16e0-441c-9ce5-796acd8551b2" path="/var/lib/kubelet/pods/156f466e-16e0-441c-9ce5-796acd8551b2/volumes" Oct 30 22:02:26.577633 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:26.577619 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e67aa41-ff98-4eab-a76c-020b1de1ff8e" path="/var/lib/kubelet/pods/1e67aa41-ff98-4eab-a76c-020b1de1ff8e/volumes" Oct 30 22:02:26.578020 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:26.578007 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f87e7e5a-9553-4dcb-9efa-d6557a171331" path="/var/lib/kubelet/pods/f87e7e5a-9553-4dcb-9efa-d6557a171331/volumes" Oct 30 22:02:32.188826 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:32.188786 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-a2dced-7f7d56769b-9fh9l" podUID="55bca164-d7e2-4f16-a24d-2aea919e3047" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:02:32.785490 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:32.785468 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-a2dced-7f7d56769b-9fh9l" Oct 30 22:02:32.904928 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:32.904862 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55bca164-d7e2-4f16-a24d-2aea919e3047-kube-api-access-9p9g4\" (UniqueName: \"kubernetes.io/projected/55bca164-d7e2-4f16-a24d-2aea919e3047-kube-api-access-9p9g4\") pod \"55bca164-d7e2-4f16-a24d-2aea919e3047\" (UID: \"55bca164-d7e2-4f16-a24d-2aea919e3047\") " Oct 30 22:02:32.905057 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:32.905043 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55bca164-d7e2-4f16-a24d-2aea919e3047-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55bca164-d7e2-4f16-a24d-2aea919e3047-plugins-config-volume\") pod \"55bca164-d7e2-4f16-a24d-2aea919e3047\" (UID: \"55bca164-d7e2-4f16-a24d-2aea919e3047\") " Oct 30 22:02:32.905477 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:32.905449 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55bca164-d7e2-4f16-a24d-2aea919e3047-plugins-config-volume" pod "55bca164-d7e2-4f16-a24d-2aea919e3047" (UID: "55bca164-d7e2-4f16-a24d-2aea919e3047"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:32.906902 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:32.906868 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55bca164-d7e2-4f16-a24d-2aea919e3047-kube-api-access-9p9g4" pod "55bca164-d7e2-4f16-a24d-2aea919e3047" (UID: "55bca164-d7e2-4f16-a24d-2aea919e3047"). InnerVolumeSpecName "kube-api-access-9p9g4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:33.006085 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:33.006054 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55bca164-d7e2-4f16-a24d-2aea919e3047-plugins-config-volume\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:33.006085 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:33.006081 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p9g4\" (UniqueName: \"kubernetes.io/projected/55bca164-d7e2-4f16-a24d-2aea919e3047-kube-api-access-9p9g4\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:33.241895 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:33.241858 4560 generic.go:356] "Generic (PLEG): container finished" podID="55bca164-d7e2-4f16-a24d-2aea919e3047" containerID="e250887e902d002a0e7f619d1ad6d1ca7b6bb53a9b4da7b021a8332d57e30444" exitCode=0 Oct 30 22:02:33.242307 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:33.241906 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-a2dced-7f7d56769b-9fh9l" Oct 30 22:02:33.242307 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:33.241943 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-a2dced-7f7d56769b-9fh9l" event={"ID":"55bca164-d7e2-4f16-a24d-2aea919e3047","Type":"ContainerDied","Data":"e250887e902d002a0e7f619d1ad6d1ca7b6bb53a9b4da7b021a8332d57e30444"} Oct 30 22:02:33.242307 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:33.242000 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-a2dced-7f7d56769b-9fh9l" event={"ID":"55bca164-d7e2-4f16-a24d-2aea919e3047","Type":"ContainerDied","Data":"3a3564b78e707edb7b321f704c9a43a605643540ebb60e252cd20aed89e51fe6"} Oct 30 22:02:33.242307 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:33.242023 4560 scope.go:122] "RemoveContainer" containerID="e250887e902d002a0e7f619d1ad6d1ca7b6bb53a9b4da7b021a8332d57e30444" Oct 30 22:02:33.251578 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:33.251560 4560 scope.go:122] "RemoveContainer" containerID="e250887e902d002a0e7f619d1ad6d1ca7b6bb53a9b4da7b021a8332d57e30444" Oct 30 22:02:33.251840 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:02:33.251822 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e250887e902d002a0e7f619d1ad6d1ca7b6bb53a9b4da7b021a8332d57e30444\": container with ID starting with e250887e902d002a0e7f619d1ad6d1ca7b6bb53a9b4da7b021a8332d57e30444 not found: ID does not exist" containerID="e250887e902d002a0e7f619d1ad6d1ca7b6bb53a9b4da7b021a8332d57e30444" Oct 30 22:02:33.251937 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:33.251849 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e250887e902d002a0e7f619d1ad6d1ca7b6bb53a9b4da7b021a8332d57e30444"} err="failed to get container status \"e250887e902d002a0e7f619d1ad6d1ca7b6bb53a9b4da7b021a8332d57e30444\": rpc error: code = NotFound desc = could not find container \"e250887e902d002a0e7f619d1ad6d1ca7b6bb53a9b4da7b021a8332d57e30444\": container with ID starting with e250887e902d002a0e7f619d1ad6d1ca7b6bb53a9b4da7b021a8332d57e30444 not found: ID does not exist" Oct 30 22:02:33.398239 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:33.398205 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a2dced-7f7d56769b-9fh9l"] Oct 30 22:02:33.402253 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:33.402226 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a2dced-7f7d56769b-9fh9l"] Oct 30 22:02:34.575741 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:34.575708 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55bca164-d7e2-4f16-a24d-2aea919e3047" path="/var/lib/kubelet/pods/55bca164-d7e2-4f16-a24d-2aea919e3047/volumes" Oct 30 22:02:36.432195 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.432161 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23f42261-z5dx2"] Oct 30 22:02:36.432604 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.432557 4560 cpu_manager.go:414] "containerMap: removing container" podUID="2eecc880-6f0d-4a2c-888e-32803ef8c86b" containerName="revoke-keys" Oct 30 22:02:36.432604 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.432569 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eecc880-6f0d-4a2c-888e-32803ef8c86b" containerName="revoke-keys" Oct 30 22:02:36.432604 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.432583 4560 cpu_manager.go:414] "containerMap: removing container" podUID="1e67aa41-ff98-4eab-a76c-020b1de1ff8e" containerName="istio-proxy" Oct 30 22:02:36.432604 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.432588 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e67aa41-ff98-4eab-a76c-020b1de1ff8e" containerName="istio-proxy" Oct 30 22:02:36.432604 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.432598 4560 cpu_manager.go:414] "containerMap: removing container" podUID="f87e7e5a-9553-4dcb-9efa-d6557a171331" containerName="istio-proxy" Oct 30 22:02:36.432604 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.432605 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f87e7e5a-9553-4dcb-9efa-d6557a171331" containerName="istio-proxy" Oct 30 22:02:36.432829 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.432616 4560 cpu_manager.go:414] "containerMap: removing container" podUID="55bca164-d7e2-4f16-a24d-2aea919e3047" containerName="payload-processing" Oct 30 22:02:36.432829 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.432622 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55bca164-d7e2-4f16-a24d-2aea919e3047" containerName="payload-processing" Oct 30 22:02:36.432829 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.432629 4560 cpu_manager.go:414] "containerMap: removing container" podUID="156f466e-16e0-441c-9ce5-796acd8551b2" containerName="istio-proxy" Oct 30 22:02:36.432829 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.432634 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="156f466e-16e0-441c-9ce5-796acd8551b2" containerName="istio-proxy" Oct 30 22:02:36.432829 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.432643 4560 cpu_manager.go:414] "containerMap: removing container" podUID="8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3" containerName="main" Oct 30 22:02:36.432829 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.432648 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3" containerName="main" Oct 30 22:02:36.432829 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.432655 4560 cpu_manager.go:414] "containerMap: removing container" podUID="45a5bac0-28ab-4c78-856e-303f6d4522de" containerName="main" Oct 30 22:02:36.432829 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.432659 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45a5bac0-28ab-4c78-856e-303f6d4522de" containerName="main" Oct 30 22:02:36.432829 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.432666 4560 cpu_manager.go:414] "containerMap: removing container" podUID="81ad1290-d768-489d-9bcf-2e8f9214b369" containerName="revoke-keys" Oct 30 22:02:36.432829 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.432670 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81ad1290-d768-489d-9bcf-2e8f9214b369" containerName="revoke-keys" Oct 30 22:02:36.432829 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.432678 4560 cpu_manager.go:414] "containerMap: removing container" podUID="b80b0fba-3830-4a74-a724-aead35443b89" containerName="main" Oct 30 22:02:36.432829 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.432682 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b80b0fba-3830-4a74-a724-aead35443b89" containerName="main" Oct 30 22:02:36.432829 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.432742 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e67aa41-ff98-4eab-a76c-020b1de1ff8e" containerName="istio-proxy" Oct 30 22:02:36.432829 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.432750 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="b80b0fba-3830-4a74-a724-aead35443b89" containerName="main" Oct 30 22:02:36.432829 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.432756 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="f87e7e5a-9553-4dcb-9efa-d6557a171331" containerName="istio-proxy" Oct 30 22:02:36.432829 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.432762 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="55bca164-d7e2-4f16-a24d-2aea919e3047" containerName="payload-processing" Oct 30 22:02:36.432829 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.432768 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="156f466e-16e0-441c-9ce5-796acd8551b2" containerName="istio-proxy" Oct 30 22:02:36.432829 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.432775 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eecc880-6f0d-4a2c-888e-32803ef8c86b" containerName="revoke-keys" Oct 30 22:02:36.432829 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.432779 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cc70a2a-65b6-4dbc-b6e1-1bedf8354fe3" containerName="main" Oct 30 22:02:36.432829 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.432784 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="45a5bac0-28ab-4c78-856e-303f6d4522de" containerName="main" Oct 30 22:02:36.432829 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.432789 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="81ad1290-d768-489d-9bcf-2e8f9214b369" containerName="revoke-keys" Oct 30 22:02:36.436898 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.436881 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f42261-z5dx2" Oct 30 22:02:36.479201 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.479171 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e289e3d-ttgz2"] Oct 30 22:02:36.482454 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.482437 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e289e3d-ttgz2" Oct 30 22:02:36.523664 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.523636 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:02:36.523664 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.523637 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:02:36.523850 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.523647 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:02:36.527641 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.527611 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23f42261-z5dx2"] Oct 30 22:02:36.538773 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.538755 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:02:36.543329 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.543307 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e289e3d-ttgz2"] Oct 30 22:02:36.553804 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.553783 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4b81eaf-72d9-4ed8-8325-52a8a31308f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23f42261-z5dx2\" (UID: \"c4b81eaf-72d9-4ed8-8325-52a8a31308f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f42261-z5dx2" Oct 30 22:02:36.553957 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.553937 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7v5g\" (UniqueName: \"kubernetes.io/projected/c4b81eaf-72d9-4ed8-8325-52a8a31308f2-kube-api-access-m7v5g\") pod \"maas-api-revoke-sub-23f42261-z5dx2\" (UID: \"c4b81eaf-72d9-4ed8-8325-52a8a31308f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f42261-z5dx2" Oct 30 22:02:36.554006 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.553996 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4b81eaf-72d9-4ed8-8325-52a8a31308f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23f42261-z5dx2\" (UID: \"c4b81eaf-72d9-4ed8-8325-52a8a31308f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f42261-z5dx2" Oct 30 22:02:36.655226 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.655193 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4b81eaf-72d9-4ed8-8325-52a8a31308f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23f42261-z5dx2\" (UID: \"c4b81eaf-72d9-4ed8-8325-52a8a31308f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f42261-z5dx2" Oct 30 22:02:36.655424 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.655256 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzptf\" (UniqueName: \"kubernetes.io/projected/d860f97c-a48b-4848-8512-79351fb19f37-kube-api-access-rzptf\") pod \"maas-api-revoke-sub-5e289e3d-ttgz2\" (UID: \"d860f97c-a48b-4848-8512-79351fb19f37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e289e3d-ttgz2" Oct 30 22:02:36.655424 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.655289 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4b81eaf-72d9-4ed8-8325-52a8a31308f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23f42261-z5dx2\" (UID: \"c4b81eaf-72d9-4ed8-8325-52a8a31308f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f42261-z5dx2" Oct 30 22:02:36.655424 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.655315 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d860f97c-a48b-4848-8512-79351fb19f37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e289e3d-ttgz2\" (UID: \"d860f97c-a48b-4848-8512-79351fb19f37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e289e3d-ttgz2" Oct 30 22:02:36.655530 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.655481 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d860f97c-a48b-4848-8512-79351fb19f37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e289e3d-ttgz2\" (UID: \"d860f97c-a48b-4848-8512-79351fb19f37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e289e3d-ttgz2" Oct 30 22:02:36.655626 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.655612 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7v5g\" (UniqueName: \"kubernetes.io/projected/c4b81eaf-72d9-4ed8-8325-52a8a31308f2-kube-api-access-m7v5g\") pod \"maas-api-revoke-sub-23f42261-z5dx2\" (UID: \"c4b81eaf-72d9-4ed8-8325-52a8a31308f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f42261-z5dx2" Oct 30 22:02:36.655847 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.655819 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4b81eaf-72d9-4ed8-8325-52a8a31308f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23f42261-z5dx2\" (UID: \"c4b81eaf-72d9-4ed8-8325-52a8a31308f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f42261-z5dx2" Oct 30 22:02:36.657660 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.657642 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4b81eaf-72d9-4ed8-8325-52a8a31308f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23f42261-z5dx2\" (UID: \"c4b81eaf-72d9-4ed8-8325-52a8a31308f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f42261-z5dx2" Oct 30 22:02:36.713700 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.713641 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7v5g\" (UniqueName: \"kubernetes.io/projected/c4b81eaf-72d9-4ed8-8325-52a8a31308f2-kube-api-access-m7v5g\") pod \"maas-api-revoke-sub-23f42261-z5dx2\" (UID: \"c4b81eaf-72d9-4ed8-8325-52a8a31308f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f42261-z5dx2" Oct 30 22:02:36.749502 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.749476 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f42261-z5dx2" Oct 30 22:02:36.756312 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.756288 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d860f97c-a48b-4848-8512-79351fb19f37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e289e3d-ttgz2\" (UID: \"d860f97c-a48b-4848-8512-79351fb19f37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e289e3d-ttgz2" Oct 30 22:02:36.756460 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.756334 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d860f97c-a48b-4848-8512-79351fb19f37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e289e3d-ttgz2\" (UID: \"d860f97c-a48b-4848-8512-79351fb19f37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e289e3d-ttgz2" Oct 30 22:02:36.756530 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.756474 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzptf\" (UniqueName: \"kubernetes.io/projected/d860f97c-a48b-4848-8512-79351fb19f37-kube-api-access-rzptf\") pod \"maas-api-revoke-sub-5e289e3d-ttgz2\" (UID: \"d860f97c-a48b-4848-8512-79351fb19f37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e289e3d-ttgz2" Oct 30 22:02:36.757095 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.757072 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d860f97c-a48b-4848-8512-79351fb19f37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e289e3d-ttgz2\" (UID: \"d860f97c-a48b-4848-8512-79351fb19f37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e289e3d-ttgz2" Oct 30 22:02:36.758549 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.758528 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d860f97c-a48b-4848-8512-79351fb19f37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e289e3d-ttgz2\" (UID: \"d860f97c-a48b-4848-8512-79351fb19f37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e289e3d-ttgz2" Oct 30 22:02:36.821031 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:36.820998 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzptf\" (UniqueName: \"kubernetes.io/projected/d860f97c-a48b-4848-8512-79351fb19f37-kube-api-access-rzptf\") pod \"maas-api-revoke-sub-5e289e3d-ttgz2\" (UID: \"d860f97c-a48b-4848-8512-79351fb19f37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e289e3d-ttgz2" Oct 30 22:02:37.094724 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:37.094691 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e289e3d-ttgz2" Oct 30 22:02:37.122785 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:37.122748 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23f42261-z5dx2"] Oct 30 22:02:37.128095 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:02:37.128067 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4b81eaf_72d9_4ed8_8325_52a8a31308f2.slice/crio-e96bca82d526a588acaa050744a5e07c89fc24675e94e9dafb4cebc65ce4eff8 WatchSource:0}: Error finding container e96bca82d526a588acaa050744a5e07c89fc24675e94e9dafb4cebc65ce4eff8: Status 404 returned error can't find the container with id e96bca82d526a588acaa050744a5e07c89fc24675e94e9dafb4cebc65ce4eff8 Oct 30 22:02:37.267784 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:37.267744 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f42261-z5dx2" event={"ID":"c4b81eaf-72d9-4ed8-8325-52a8a31308f2","Type":"ContainerStarted","Data":"e96bca82d526a588acaa050744a5e07c89fc24675e94e9dafb4cebc65ce4eff8"} Oct 30 22:02:37.473039 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:37.473004 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e289e3d-ttgz2"] Oct 30 22:02:37.478522 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:02:37.478495 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd860f97c_a48b_4848_8512_79351fb19f37.slice/crio-aa1fba947828470a5819036fd0471662ce46a038096918503ab89bf10c928463 WatchSource:0}: Error finding container aa1fba947828470a5819036fd0471662ce46a038096918503ab89bf10c928463: Status 404 returned error can't find the container with id aa1fba947828470a5819036fd0471662ce46a038096918503ab89bf10c928463 Oct 30 22:02:38.279181 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:38.279142 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e289e3d-ttgz2" event={"ID":"d860f97c-a48b-4848-8512-79351fb19f37","Type":"ContainerStarted","Data":"504a74affb73311e04fa9e0ecf28fb73a847493022bb895795490add813769a7"} Oct 30 22:02:38.279367 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:38.279197 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e289e3d-ttgz2" event={"ID":"d860f97c-a48b-4848-8512-79351fb19f37","Type":"ContainerStarted","Data":"aa1fba947828470a5819036fd0471662ce46a038096918503ab89bf10c928463"} Oct 30 22:02:38.280770 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:38.280745 4560 generic.go:356] "Generic (PLEG): container finished" podID="c4b81eaf-72d9-4ed8-8325-52a8a31308f2" containerID="5fd1675b885c90c47450267748e6b5d0547212550fba987c98b6841b4b8c68d4" exitCode=0 Oct 30 22:02:38.280906 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:38.280825 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f42261-z5dx2" event={"ID":"c4b81eaf-72d9-4ed8-8325-52a8a31308f2","Type":"ContainerDied","Data":"5fd1675b885c90c47450267748e6b5d0547212550fba987c98b6841b4b8c68d4"} Oct 30 22:02:38.402227 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:38.401924 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e289e3d-ttgz2" podStartSLOduration=2.401889352 podStartE2EDuration="2.401889352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:38.401211854 +0000 UTC m=+2933.955911632" watchObservedRunningTime="2026-10-30 22:02:38.401889352 +0000 UTC m=+2933.956589127" Oct 30 22:02:39.290548 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:39.290510 4560 generic.go:356] "Generic (PLEG): container finished" podID="d860f97c-a48b-4848-8512-79351fb19f37" containerID="504a74affb73311e04fa9e0ecf28fb73a847493022bb895795490add813769a7" exitCode=0 Oct 30 22:02:39.291047 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:39.290594 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e289e3d-ttgz2" event={"ID":"d860f97c-a48b-4848-8512-79351fb19f37","Type":"ContainerDied","Data":"504a74affb73311e04fa9e0ecf28fb73a847493022bb895795490add813769a7"} Oct 30 22:02:39.480894 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:39.480866 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f42261-z5dx2" Oct 30 22:02:39.590245 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:39.590157 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4b81eaf-72d9-4ed8-8325-52a8a31308f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4b81eaf-72d9-4ed8-8325-52a8a31308f2-maas-api-xks-ca\") pod \"c4b81eaf-72d9-4ed8-8325-52a8a31308f2\" (UID: \"c4b81eaf-72d9-4ed8-8325-52a8a31308f2\") " Oct 30 22:02:39.590245 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:39.590224 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4b81eaf-72d9-4ed8-8325-52a8a31308f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4b81eaf-72d9-4ed8-8325-52a8a31308f2-maas-api-openshift-ca\") pod \"c4b81eaf-72d9-4ed8-8325-52a8a31308f2\" (UID: \"c4b81eaf-72d9-4ed8-8325-52a8a31308f2\") " Oct 30 22:02:39.590507 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:39.590316 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4b81eaf-72d9-4ed8-8325-52a8a31308f2-kube-api-access-m7v5g\" (UniqueName: \"kubernetes.io/projected/c4b81eaf-72d9-4ed8-8325-52a8a31308f2-kube-api-access-m7v5g\") pod \"c4b81eaf-72d9-4ed8-8325-52a8a31308f2\" (UID: \"c4b81eaf-72d9-4ed8-8325-52a8a31308f2\") " Oct 30 22:02:39.590668 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:39.590632 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4b81eaf-72d9-4ed8-8325-52a8a31308f2-maas-api-openshift-ca" pod "c4b81eaf-72d9-4ed8-8325-52a8a31308f2" (UID: "c4b81eaf-72d9-4ed8-8325-52a8a31308f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:39.590811 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:39.590755 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4b81eaf-72d9-4ed8-8325-52a8a31308f2-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:39.592942 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:39.592897 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4b81eaf-72d9-4ed8-8325-52a8a31308f2-kube-api-access-m7v5g" pod "c4b81eaf-72d9-4ed8-8325-52a8a31308f2" (UID: "c4b81eaf-72d9-4ed8-8325-52a8a31308f2"). InnerVolumeSpecName "kube-api-access-m7v5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:39.594491 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:39.594466 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4b81eaf-72d9-4ed8-8325-52a8a31308f2-maas-api-xks-ca" pod "c4b81eaf-72d9-4ed8-8325-52a8a31308f2" (UID: "c4b81eaf-72d9-4ed8-8325-52a8a31308f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:39.691623 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:39.691585 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4b81eaf-72d9-4ed8-8325-52a8a31308f2-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:39.691623 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:39.691618 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7v5g\" (UniqueName: \"kubernetes.io/projected/c4b81eaf-72d9-4ed8-8325-52a8a31308f2-kube-api-access-m7v5g\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:40.299205 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:40.299166 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f42261-z5dx2" event={"ID":"c4b81eaf-72d9-4ed8-8325-52a8a31308f2","Type":"ContainerDied","Data":"e96bca82d526a588acaa050744a5e07c89fc24675e94e9dafb4cebc65ce4eff8"} Oct 30 22:02:40.299692 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:40.299231 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e96bca82d526a588acaa050744a5e07c89fc24675e94e9dafb4cebc65ce4eff8" Oct 30 22:02:40.299692 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:40.299176 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f42261-z5dx2" Oct 30 22:02:40.429830 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:40.429805 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e289e3d-ttgz2" Oct 30 22:02:40.599887 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:40.599817 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d860f97c-a48b-4848-8512-79351fb19f37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d860f97c-a48b-4848-8512-79351fb19f37-maas-api-xks-ca\") pod \"d860f97c-a48b-4848-8512-79351fb19f37\" (UID: \"d860f97c-a48b-4848-8512-79351fb19f37\") " Oct 30 22:02:40.600036 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:40.599906 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d860f97c-a48b-4848-8512-79351fb19f37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d860f97c-a48b-4848-8512-79351fb19f37-maas-api-openshift-ca\") pod \"d860f97c-a48b-4848-8512-79351fb19f37\" (UID: \"d860f97c-a48b-4848-8512-79351fb19f37\") " Oct 30 22:02:40.600036 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:40.599977 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d860f97c-a48b-4848-8512-79351fb19f37-kube-api-access-rzptf\" (UniqueName: \"kubernetes.io/projected/d860f97c-a48b-4848-8512-79351fb19f37-kube-api-access-rzptf\") pod \"d860f97c-a48b-4848-8512-79351fb19f37\" (UID: \"d860f97c-a48b-4848-8512-79351fb19f37\") " Oct 30 22:02:40.600316 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:40.600292 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d860f97c-a48b-4848-8512-79351fb19f37-maas-api-openshift-ca" pod "d860f97c-a48b-4848-8512-79351fb19f37" (UID: "d860f97c-a48b-4848-8512-79351fb19f37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:40.601813 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:40.601789 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d860f97c-a48b-4848-8512-79351fb19f37-maas-api-xks-ca" pod "d860f97c-a48b-4848-8512-79351fb19f37" (UID: "d860f97c-a48b-4848-8512-79351fb19f37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:40.601946 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:40.601926 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d860f97c-a48b-4848-8512-79351fb19f37-kube-api-access-rzptf" pod "d860f97c-a48b-4848-8512-79351fb19f37" (UID: "d860f97c-a48b-4848-8512-79351fb19f37"). InnerVolumeSpecName "kube-api-access-rzptf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:40.701397 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:40.701363 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzptf\" (UniqueName: \"kubernetes.io/projected/d860f97c-a48b-4848-8512-79351fb19f37-kube-api-access-rzptf\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:40.701397 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:40.701397 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d860f97c-a48b-4848-8512-79351fb19f37-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:40.701636 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:40.701412 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d860f97c-a48b-4848-8512-79351fb19f37-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:41.307068 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:41.307039 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e289e3d-ttgz2" Oct 30 22:02:41.307586 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:41.307085 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e289e3d-ttgz2" event={"ID":"d860f97c-a48b-4848-8512-79351fb19f37","Type":"ContainerDied","Data":"aa1fba947828470a5819036fd0471662ce46a038096918503ab89bf10c928463"} Oct 30 22:02:41.307586 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:41.307140 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa1fba947828470a5819036fd0471662ce46a038096918503ab89bf10c928463" Oct 30 22:02:43.258132 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:43.258092 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a2dced-pbwv6"] Oct 30 22:02:43.304278 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:43.304241 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a2dced-pbwv6"] Oct 30 22:02:44.575315 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:44.575274 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="636e1507-92bf-48ea-b1c5-b93bc4d08f66" path="/var/lib/kubelet/pods/636e1507-92bf-48ea-b1c5-b93bc4d08f66/volumes" Oct 30 22:02:47.407409 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:47.407375 4560 scope.go:122] "RemoveContainer" containerID="a6da00c9b2007936b529896fe61a57bfab83a3edabc837e94296ba99306a37df" Oct 30 22:02:47.416482 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:47.416460 4560 scope.go:122] "RemoveContainer" containerID="6639cbbd016b9d7d69eb3d4a5bcb585552aeae0cb56eb8c5d0c1c57e0bacc78e" Oct 30 22:02:49.719776 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:02:49.719740 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:02:52.047171 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:52.047135 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw"] Oct 30 22:02:52.047568 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:52.047358 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" podUID="b3a33e86-072f-458f-837d-d349fb6981da" containerName="istio-proxy" containerID="cri-o://053382ed30d17e0dafc5129ff6221cd1b0f790b6ecad7af1c52b219cef88b5e2" gracePeriod=30 Oct 30 22:02:54.406412 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:54.406370 4560 patch_prober.go:29] interesting pod/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:02:54.406944 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:54.406436 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" podUID="b3a33e86-072f-458f-837d-d349fb6981da" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:02:57.310836 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:57.310810 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" Oct 30 22:02:57.436476 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:57.436400 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3a33e86-072f-458f-837d-d349fb6981da-istio-token\" (UniqueName: \"kubernetes.io/projected/b3a33e86-072f-458f-837d-d349fb6981da-istio-token\") pod \"b3a33e86-072f-458f-837d-d349fb6981da\" (UID: \"b3a33e86-072f-458f-837d-d349fb6981da\") " Oct 30 22:02:57.436628 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:57.436503 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3a33e86-072f-458f-837d-d349fb6981da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3a33e86-072f-458f-837d-d349fb6981da-istio-envoy\") pod \"b3a33e86-072f-458f-837d-d349fb6981da\" (UID: \"b3a33e86-072f-458f-837d-d349fb6981da\") " Oct 30 22:02:57.436628 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:57.436533 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3a33e86-072f-458f-837d-d349fb6981da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3a33e86-072f-458f-837d-d349fb6981da-istiod-ca-cert\") pod \"b3a33e86-072f-458f-837d-d349fb6981da\" (UID: \"b3a33e86-072f-458f-837d-d349fb6981da\") " Oct 30 22:02:57.436628 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:57.436575 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3a33e86-072f-458f-837d-d349fb6981da-kube-api-access-276gf\" (UniqueName: \"kubernetes.io/projected/b3a33e86-072f-458f-837d-d349fb6981da-kube-api-access-276gf\") pod \"b3a33e86-072f-458f-837d-d349fb6981da\" (UID: \"b3a33e86-072f-458f-837d-d349fb6981da\") " Oct 30 22:02:57.436791 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:57.436680 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3a33e86-072f-458f-837d-d349fb6981da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3a33e86-072f-458f-837d-d349fb6981da-workload-certs\") pod \"b3a33e86-072f-458f-837d-d349fb6981da\" (UID: \"b3a33e86-072f-458f-837d-d349fb6981da\") " Oct 30 22:02:57.436791 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:57.436744 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3a33e86-072f-458f-837d-d349fb6981da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3a33e86-072f-458f-837d-d349fb6981da-istio-data\") pod \"b3a33e86-072f-458f-837d-d349fb6981da\" (UID: \"b3a33e86-072f-458f-837d-d349fb6981da\") " Oct 30 22:02:57.436900 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:57.436795 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3a33e86-072f-458f-837d-d349fb6981da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3a33e86-072f-458f-837d-d349fb6981da-workload-socket\") pod \"b3a33e86-072f-458f-837d-d349fb6981da\" (UID: \"b3a33e86-072f-458f-837d-d349fb6981da\") " Oct 30 22:02:57.436960 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:57.436926 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3a33e86-072f-458f-837d-d349fb6981da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3a33e86-072f-458f-837d-d349fb6981da-istio-podinfo\") pod \"b3a33e86-072f-458f-837d-d349fb6981da\" (UID: \"b3a33e86-072f-458f-837d-d349fb6981da\") " Oct 30 22:02:57.437029 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:57.436960 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3a33e86-072f-458f-837d-d349fb6981da-istio-data" pod "b3a33e86-072f-458f-837d-d349fb6981da" (UID: "b3a33e86-072f-458f-837d-d349fb6981da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:57.437029 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:57.436971 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3a33e86-072f-458f-837d-d349fb6981da-workload-certs" pod "b3a33e86-072f-458f-837d-d349fb6981da" (UID: "b3a33e86-072f-458f-837d-d349fb6981da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:57.437029 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:57.436968 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3a33e86-072f-458f-837d-d349fb6981da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3a33e86-072f-458f-837d-d349fb6981da-credential-socket\") pod \"b3a33e86-072f-458f-837d-d349fb6981da\" (UID: \"b3a33e86-072f-458f-837d-d349fb6981da\") " Oct 30 22:02:57.437029 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:57.437002 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3a33e86-072f-458f-837d-d349fb6981da-istiod-ca-cert" pod "b3a33e86-072f-458f-837d-d349fb6981da" (UID: "b3a33e86-072f-458f-837d-d349fb6981da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:57.437263 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:57.437149 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3a33e86-072f-458f-837d-d349fb6981da-credential-socket" pod "b3a33e86-072f-458f-837d-d349fb6981da" (UID: "b3a33e86-072f-458f-837d-d349fb6981da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:57.437263 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:57.437161 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3a33e86-072f-458f-837d-d349fb6981da-workload-socket" pod "b3a33e86-072f-458f-837d-d349fb6981da" (UID: "b3a33e86-072f-458f-837d-d349fb6981da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:57.437454 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:57.437434 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3a33e86-072f-458f-837d-d349fb6981da-istio-data\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:57.437519 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:57.437456 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3a33e86-072f-458f-837d-d349fb6981da-workload-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:57.437519 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:57.437466 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3a33e86-072f-458f-837d-d349fb6981da-credential-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:57.437519 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:57.437475 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3a33e86-072f-458f-837d-d349fb6981da-istiod-ca-cert\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:57.437519 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:57.437490 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3a33e86-072f-458f-837d-d349fb6981da-workload-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:57.438939 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:57.438915 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3a33e86-072f-458f-837d-d349fb6981da-istio-envoy" pod "b3a33e86-072f-458f-837d-d349fb6981da" (UID: "b3a33e86-072f-458f-837d-d349fb6981da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:57.439073 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:57.438948 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3a33e86-072f-458f-837d-d349fb6981da-istio-podinfo" pod "b3a33e86-072f-458f-837d-d349fb6981da" (UID: "b3a33e86-072f-458f-837d-d349fb6981da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:02:57.439261 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:57.439244 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3a33e86-072f-458f-837d-d349fb6981da-kube-api-access-276gf" pod "b3a33e86-072f-458f-837d-d349fb6981da" (UID: "b3a33e86-072f-458f-837d-d349fb6981da"). InnerVolumeSpecName "kube-api-access-276gf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:57.439261 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:57.439253 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3a33e86-072f-458f-837d-d349fb6981da-istio-token" pod "b3a33e86-072f-458f-837d-d349fb6981da" (UID: "b3a33e86-072f-458f-837d-d349fb6981da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:57.450493 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:57.450463 4560 generic.go:356] "Generic (PLEG): container finished" podID="b3a33e86-072f-458f-837d-d349fb6981da" containerID="053382ed30d17e0dafc5129ff6221cd1b0f790b6ecad7af1c52b219cef88b5e2" exitCode=0 Oct 30 22:02:57.450614 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:57.450513 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" Oct 30 22:02:57.450614 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:57.450544 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" event={"ID":"b3a33e86-072f-458f-837d-d349fb6981da","Type":"ContainerDied","Data":"053382ed30d17e0dafc5129ff6221cd1b0f790b6ecad7af1c52b219cef88b5e2"} Oct 30 22:02:57.450614 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:57.450599 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw" event={"ID":"b3a33e86-072f-458f-837d-d349fb6981da","Type":"ContainerDied","Data":"3d4f4e2864c04ae563d54834613fdd209da8583cc675424725fdde4acdd660f8"} Oct 30 22:02:57.450804 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:57.450627 4560 scope.go:122] "RemoveContainer" containerID="053382ed30d17e0dafc5129ff6221cd1b0f790b6ecad7af1c52b219cef88b5e2" Oct 30 22:02:57.461787 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:57.461767 4560 scope.go:122] "RemoveContainer" containerID="053382ed30d17e0dafc5129ff6221cd1b0f790b6ecad7af1c52b219cef88b5e2" Oct 30 22:02:57.462064 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:02:57.462045 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"053382ed30d17e0dafc5129ff6221cd1b0f790b6ecad7af1c52b219cef88b5e2\": container with ID starting with 053382ed30d17e0dafc5129ff6221cd1b0f790b6ecad7af1c52b219cef88b5e2 not found: ID does not exist" containerID="053382ed30d17e0dafc5129ff6221cd1b0f790b6ecad7af1c52b219cef88b5e2" Oct 30 22:02:57.462131 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:57.462077 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"053382ed30d17e0dafc5129ff6221cd1b0f790b6ecad7af1c52b219cef88b5e2"} err="failed to get container status \"053382ed30d17e0dafc5129ff6221cd1b0f790b6ecad7af1c52b219cef88b5e2\": rpc error: code = NotFound desc = could not find container \"053382ed30d17e0dafc5129ff6221cd1b0f790b6ecad7af1c52b219cef88b5e2\": container with ID starting with 053382ed30d17e0dafc5129ff6221cd1b0f790b6ecad7af1c52b219cef88b5e2 not found: ID does not exist" Oct 30 22:02:57.538707 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:57.538676 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3a33e86-072f-458f-837d-d349fb6981da-istio-podinfo\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:57.538873 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:57.538722 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3a33e86-072f-458f-837d-d349fb6981da-istio-token\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:57.538873 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:57.538733 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3a33e86-072f-458f-837d-d349fb6981da-istio-envoy\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:57.538873 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:57.538745 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-276gf\" (UniqueName: \"kubernetes.io/projected/b3a33e86-072f-458f-837d-d349fb6981da-kube-api-access-276gf\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:57.592794 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:57.592759 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw"] Oct 30 22:02:57.613771 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:57.613742 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a2dced-openshift-default-557799c76c-7t4rw"] Oct 30 22:02:58.575666 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:02:58.575632 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3a33e86-072f-458f-837d-d349fb6981da" path="/var/lib/kubelet/pods/b3a33e86-072f-458f-837d-d349fb6981da/volumes" Oct 30 22:03:05.438136 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:05.438102 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-28c192-q5tc2"] Oct 30 22:03:05.438564 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:05.438544 4560 cpu_manager.go:414] "containerMap: removing container" podUID="c4b81eaf-72d9-4ed8-8325-52a8a31308f2" containerName="revoke-keys" Oct 30 22:03:05.438610 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:05.438565 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4b81eaf-72d9-4ed8-8325-52a8a31308f2" containerName="revoke-keys" Oct 30 22:03:05.438610 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:05.438577 4560 cpu_manager.go:414] "containerMap: removing container" podUID="b3a33e86-072f-458f-837d-d349fb6981da" containerName="istio-proxy" Oct 30 22:03:05.438610 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:05.438583 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3a33e86-072f-458f-837d-d349fb6981da" containerName="istio-proxy" Oct 30 22:03:05.438610 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:05.438602 4560 cpu_manager.go:414] "containerMap: removing container" podUID="d860f97c-a48b-4848-8512-79351fb19f37" containerName="revoke-keys" Oct 30 22:03:05.438610 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:05.438607 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d860f97c-a48b-4848-8512-79351fb19f37" containerName="revoke-keys" Oct 30 22:03:05.438777 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:05.438692 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="d860f97c-a48b-4848-8512-79351fb19f37" containerName="revoke-keys" Oct 30 22:03:05.438777 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:05.438704 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4b81eaf-72d9-4ed8-8325-52a8a31308f2" containerName="revoke-keys" Oct 30 22:03:05.438777 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:05.438715 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3a33e86-072f-458f-837d-d349fb6981da" containerName="istio-proxy" Oct 30 22:03:05.443156 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:05.443137 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-28c192-q5tc2" Oct 30 22:03:05.496946 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:05.496912 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-28c192-q5tc2"] Oct 30 22:03:05.522814 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:05.522781 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80066c33-43b8-4393-af50-10b6a87e84f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-28c192-q5tc2\" (UID: \"80066c33-43b8-4393-af50-10b6a87e84f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-28c192-q5tc2" Oct 30 22:03:05.522970 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:05.522939 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fjdh\" (UniqueName: \"kubernetes.io/projected/80066c33-43b8-4393-af50-10b6a87e84f3-kube-api-access-2fjdh\") pod \"maas-api-revoke-keys-e2e-api-a-28c192-q5tc2\" (UID: \"80066c33-43b8-4393-af50-10b6a87e84f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-28c192-q5tc2" Oct 30 22:03:05.523106 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:05.523073 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80066c33-43b8-4393-af50-10b6a87e84f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-28c192-q5tc2\" (UID: \"80066c33-43b8-4393-af50-10b6a87e84f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-28c192-q5tc2" Oct 30 22:03:05.540088 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:05.540061 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:03:05.540232 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:05.540113 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:03:05.540232 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:05.540140 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:03:05.540232 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:05.540158 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:03:05.623893 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:05.623861 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80066c33-43b8-4393-af50-10b6a87e84f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-28c192-q5tc2\" (UID: \"80066c33-43b8-4393-af50-10b6a87e84f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-28c192-q5tc2" Oct 30 22:03:05.624078 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:05.623914 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80066c33-43b8-4393-af50-10b6a87e84f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-28c192-q5tc2\" (UID: \"80066c33-43b8-4393-af50-10b6a87e84f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-28c192-q5tc2" Oct 30 22:03:05.624078 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:05.624042 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fjdh\" (UniqueName: \"kubernetes.io/projected/80066c33-43b8-4393-af50-10b6a87e84f3-kube-api-access-2fjdh\") pod \"maas-api-revoke-keys-e2e-api-a-28c192-q5tc2\" (UID: \"80066c33-43b8-4393-af50-10b6a87e84f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-28c192-q5tc2" Oct 30 22:03:05.624548 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:05.624527 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80066c33-43b8-4393-af50-10b6a87e84f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-28c192-q5tc2\" (UID: \"80066c33-43b8-4393-af50-10b6a87e84f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-28c192-q5tc2" Oct 30 22:03:05.626293 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:05.626277 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80066c33-43b8-4393-af50-10b6a87e84f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-28c192-q5tc2\" (UID: \"80066c33-43b8-4393-af50-10b6a87e84f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-28c192-q5tc2" Oct 30 22:03:05.682194 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:05.682164 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fjdh\" (UniqueName: \"kubernetes.io/projected/80066c33-43b8-4393-af50-10b6a87e84f3-kube-api-access-2fjdh\") pod \"maas-api-revoke-keys-e2e-api-a-28c192-q5tc2\" (UID: \"80066c33-43b8-4393-af50-10b6a87e84f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-28c192-q5tc2" Oct 30 22:03:05.755868 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:05.755839 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-28c192-q5tc2" Oct 30 22:03:06.129110 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:06.129029 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-28c192-q5tc2"] Oct 30 22:03:06.134759 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:03:06.134726 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80066c33_43b8_4393_af50_10b6a87e84f3.slice/crio-4d4e9276f8b1686fd1e37c40f7f51edf27e06a8ab1ddc2ab24d8e5dccd230a43 WatchSource:0}: Error finding container 4d4e9276f8b1686fd1e37c40f7f51edf27e06a8ab1ddc2ab24d8e5dccd230a43: Status 404 returned error can't find the container with id 4d4e9276f8b1686fd1e37c40f7f51edf27e06a8ab1ddc2ab24d8e5dccd230a43 Oct 30 22:03:06.505467 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:06.505438 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-28c192-q5tc2" event={"ID":"80066c33-43b8-4393-af50-10b6a87e84f3","Type":"ContainerStarted","Data":"99b2ab641d1ccb702e1f1b407d4dc623da12089d8ba78417441f0249c878cdb1"} Oct 30 22:03:06.505821 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:06.505484 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-28c192-q5tc2" event={"ID":"80066c33-43b8-4393-af50-10b6a87e84f3","Type":"ContainerStarted","Data":"4d4e9276f8b1686fd1e37c40f7f51edf27e06a8ab1ddc2ab24d8e5dccd230a43"} Oct 30 22:03:06.620521 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:06.620470 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-28c192-q5tc2" podStartSLOduration=1.6204557720000001 podStartE2EDuration="1.620455772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:06.619427509 +0000 UTC m=+2962.174127287" watchObservedRunningTime="2026-10-30 22:03:06.620455772 +0000 UTC m=+2962.175155550" Oct 30 22:03:07.521649 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:07.521610 4560 generic.go:356] "Generic (PLEG): container finished" podID="80066c33-43b8-4393-af50-10b6a87e84f3" containerID="99b2ab641d1ccb702e1f1b407d4dc623da12089d8ba78417441f0249c878cdb1" exitCode=0 Oct 30 22:03:07.522370 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:07.522116 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-28c192-q5tc2" event={"ID":"80066c33-43b8-4393-af50-10b6a87e84f3","Type":"ContainerDied","Data":"99b2ab641d1ccb702e1f1b407d4dc623da12089d8ba78417441f0249c878cdb1"} Oct 30 22:03:08.693862 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:08.693841 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-28c192-q5tc2" Oct 30 22:03:08.745887 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:08.745856 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-991335-dplhn"] Oct 30 22:03:08.792762 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:08.792679 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-991335-dplhn"] Oct 30 22:03:08.857607 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:08.857574 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80066c33-43b8-4393-af50-10b6a87e84f3-kube-api-access-2fjdh\" (UniqueName: \"kubernetes.io/projected/80066c33-43b8-4393-af50-10b6a87e84f3-kube-api-access-2fjdh\") pod \"80066c33-43b8-4393-af50-10b6a87e84f3\" (UID: \"80066c33-43b8-4393-af50-10b6a87e84f3\") " Oct 30 22:03:08.857779 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:08.857614 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80066c33-43b8-4393-af50-10b6a87e84f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80066c33-43b8-4393-af50-10b6a87e84f3-maas-api-xks-ca\") pod \"80066c33-43b8-4393-af50-10b6a87e84f3\" (UID: \"80066c33-43b8-4393-af50-10b6a87e84f3\") " Oct 30 22:03:08.857779 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:08.857753 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80066c33-43b8-4393-af50-10b6a87e84f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80066c33-43b8-4393-af50-10b6a87e84f3-maas-api-openshift-ca\") pod \"80066c33-43b8-4393-af50-10b6a87e84f3\" (UID: \"80066c33-43b8-4393-af50-10b6a87e84f3\") " Oct 30 22:03:08.858105 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:08.858082 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80066c33-43b8-4393-af50-10b6a87e84f3-maas-api-openshift-ca" pod "80066c33-43b8-4393-af50-10b6a87e84f3" (UID: "80066c33-43b8-4393-af50-10b6a87e84f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:08.859671 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:08.859640 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80066c33-43b8-4393-af50-10b6a87e84f3-maas-api-xks-ca" pod "80066c33-43b8-4393-af50-10b6a87e84f3" (UID: "80066c33-43b8-4393-af50-10b6a87e84f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:08.859772 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:08.859709 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80066c33-43b8-4393-af50-10b6a87e84f3-kube-api-access-2fjdh" pod "80066c33-43b8-4393-af50-10b6a87e84f3" (UID: "80066c33-43b8-4393-af50-10b6a87e84f3"). InnerVolumeSpecName "kube-api-access-2fjdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:08.958692 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:08.958658 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80066c33-43b8-4393-af50-10b6a87e84f3-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:08.958692 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:08.958691 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fjdh\" (UniqueName: \"kubernetes.io/projected/80066c33-43b8-4393-af50-10b6a87e84f3-kube-api-access-2fjdh\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:08.958907 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:08.958706 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80066c33-43b8-4393-af50-10b6a87e84f3-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:09.536723 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:09.536683 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-28c192-q5tc2" event={"ID":"80066c33-43b8-4393-af50-10b6a87e84f3","Type":"ContainerDied","Data":"4d4e9276f8b1686fd1e37c40f7f51edf27e06a8ab1ddc2ab24d8e5dccd230a43"} Oct 30 22:03:09.536723 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:09.536707 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-28c192-q5tc2" Oct 30 22:03:09.536997 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:09.536762 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d4e9276f8b1686fd1e37c40f7f51edf27e06a8ab1ddc2ab24d8e5dccd230a43" Oct 30 22:03:10.574749 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:10.574716 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="404224b2-3856-49b0-a6fe-f28335f2580c" path="/var/lib/kubelet/pods/404224b2-3856-49b0-a6fe-f28335f2580c/volumes" Oct 30 22:03:14.049630 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:14.049596 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a02f114-r5qqf"] Oct 30 22:03:14.096390 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:14.096357 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a02f114-r5qqf"] Oct 30 22:03:14.574847 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:14.574814 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae6a851b-aa85-4993-a7d9-134b91a6ad05" path="/var/lib/kubelet/pods/ae6a851b-aa85-4993-a7d9-134b91a6ad05/volumes" Oct 30 22:03:16.332167 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:16.332132 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw"] Oct 30 22:03:16.332709 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:16.332459 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" podUID="5d83608f-bcb0-4aab-b14e-29846561a3d3" containerName="istio-proxy" containerID="cri-o://eec07909c0ab2a065e177d21c7cae12a71d37a6aba1c35630a172459f2a204d8" gracePeriod=30 Oct 30 22:03:19.683646 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:03:19.683606 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:03:21.579828 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:21.579804 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" Oct 30 22:03:21.619630 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:21.619574 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d83608f-bcb0-4aab-b14e-29846561a3d3-istio-token\" (UniqueName: \"kubernetes.io/projected/5d83608f-bcb0-4aab-b14e-29846561a3d3-istio-token\") pod \"5d83608f-bcb0-4aab-b14e-29846561a3d3\" (UID: \"5d83608f-bcb0-4aab-b14e-29846561a3d3\") " Oct 30 22:03:21.619630 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:21.619609 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d83608f-bcb0-4aab-b14e-29846561a3d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d83608f-bcb0-4aab-b14e-29846561a3d3-istio-envoy\") pod \"5d83608f-bcb0-4aab-b14e-29846561a3d3\" (UID: \"5d83608f-bcb0-4aab-b14e-29846561a3d3\") " Oct 30 22:03:21.619796 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:21.619642 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d83608f-bcb0-4aab-b14e-29846561a3d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d83608f-bcb0-4aab-b14e-29846561a3d3-workload-socket\") pod \"5d83608f-bcb0-4aab-b14e-29846561a3d3\" (UID: \"5d83608f-bcb0-4aab-b14e-29846561a3d3\") " Oct 30 22:03:21.619796 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:21.619750 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d83608f-bcb0-4aab-b14e-29846561a3d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d83608f-bcb0-4aab-b14e-29846561a3d3-credential-socket\") pod \"5d83608f-bcb0-4aab-b14e-29846561a3d3\" (UID: \"5d83608f-bcb0-4aab-b14e-29846561a3d3\") " Oct 30 22:03:21.619796 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:21.619786 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d83608f-bcb0-4aab-b14e-29846561a3d3-kube-api-access-nkpsq\" (UniqueName: \"kubernetes.io/projected/5d83608f-bcb0-4aab-b14e-29846561a3d3-kube-api-access-nkpsq\") pod \"5d83608f-bcb0-4aab-b14e-29846561a3d3\" (UID: \"5d83608f-bcb0-4aab-b14e-29846561a3d3\") " Oct 30 22:03:21.619942 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:21.619811 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d83608f-bcb0-4aab-b14e-29846561a3d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d83608f-bcb0-4aab-b14e-29846561a3d3-istio-data\") pod \"5d83608f-bcb0-4aab-b14e-29846561a3d3\" (UID: \"5d83608f-bcb0-4aab-b14e-29846561a3d3\") " Oct 30 22:03:21.619942 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:21.619896 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d83608f-bcb0-4aab-b14e-29846561a3d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d83608f-bcb0-4aab-b14e-29846561a3d3-workload-certs\") pod \"5d83608f-bcb0-4aab-b14e-29846561a3d3\" (UID: \"5d83608f-bcb0-4aab-b14e-29846561a3d3\") " Oct 30 22:03:21.620058 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:21.619942 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d83608f-bcb0-4aab-b14e-29846561a3d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d83608f-bcb0-4aab-b14e-29846561a3d3-istiod-ca-cert\") pod \"5d83608f-bcb0-4aab-b14e-29846561a3d3\" (UID: \"5d83608f-bcb0-4aab-b14e-29846561a3d3\") " Oct 30 22:03:21.620058 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:21.619926 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d83608f-bcb0-4aab-b14e-29846561a3d3-workload-socket" pod "5d83608f-bcb0-4aab-b14e-29846561a3d3" (UID: "5d83608f-bcb0-4aab-b14e-29846561a3d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:21.620058 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:21.619982 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d83608f-bcb0-4aab-b14e-29846561a3d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d83608f-bcb0-4aab-b14e-29846561a3d3-istio-podinfo\") pod \"5d83608f-bcb0-4aab-b14e-29846561a3d3\" (UID: \"5d83608f-bcb0-4aab-b14e-29846561a3d3\") " Oct 30 22:03:21.620058 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:21.619985 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d83608f-bcb0-4aab-b14e-29846561a3d3-credential-socket" pod "5d83608f-bcb0-4aab-b14e-29846561a3d3" (UID: "5d83608f-bcb0-4aab-b14e-29846561a3d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:21.620294 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:21.620163 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d83608f-bcb0-4aab-b14e-29846561a3d3-workload-certs" pod "5d83608f-bcb0-4aab-b14e-29846561a3d3" (UID: "5d83608f-bcb0-4aab-b14e-29846561a3d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:21.620385 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:21.620290 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d83608f-bcb0-4aab-b14e-29846561a3d3-istiod-ca-cert" pod "5d83608f-bcb0-4aab-b14e-29846561a3d3" (UID: "5d83608f-bcb0-4aab-b14e-29846561a3d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:21.620475 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:21.620126 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d83608f-bcb0-4aab-b14e-29846561a3d3-istio-data" pod "5d83608f-bcb0-4aab-b14e-29846561a3d3" (UID: "5d83608f-bcb0-4aab-b14e-29846561a3d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:21.620542 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:21.620459 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d83608f-bcb0-4aab-b14e-29846561a3d3-workload-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:21.620542 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:21.620506 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d83608f-bcb0-4aab-b14e-29846561a3d3-credential-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:21.620542 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:21.620527 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d83608f-bcb0-4aab-b14e-29846561a3d3-workload-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:21.620661 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:21.620542 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d83608f-bcb0-4aab-b14e-29846561a3d3-istiod-ca-cert\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:21.622042 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:21.622014 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d83608f-bcb0-4aab-b14e-29846561a3d3-kube-api-access-nkpsq" pod "5d83608f-bcb0-4aab-b14e-29846561a3d3" (UID: "5d83608f-bcb0-4aab-b14e-29846561a3d3"). InnerVolumeSpecName "kube-api-access-nkpsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:21.622294 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:21.622272 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d83608f-bcb0-4aab-b14e-29846561a3d3-istio-podinfo" pod "5d83608f-bcb0-4aab-b14e-29846561a3d3" (UID: "5d83608f-bcb0-4aab-b14e-29846561a3d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:03:21.622406 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:21.622299 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d83608f-bcb0-4aab-b14e-29846561a3d3-istio-token" pod "5d83608f-bcb0-4aab-b14e-29846561a3d3" (UID: "5d83608f-bcb0-4aab-b14e-29846561a3d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:21.622406 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:21.622322 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d83608f-bcb0-4aab-b14e-29846561a3d3-istio-envoy" pod "5d83608f-bcb0-4aab-b14e-29846561a3d3" (UID: "5d83608f-bcb0-4aab-b14e-29846561a3d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:21.630002 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:21.629979 4560 generic.go:356] "Generic (PLEG): container finished" podID="5d83608f-bcb0-4aab-b14e-29846561a3d3" containerID="eec07909c0ab2a065e177d21c7cae12a71d37a6aba1c35630a172459f2a204d8" exitCode=0 Oct 30 22:03:21.630101 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:21.630033 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" Oct 30 22:03:21.630163 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:21.630025 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" event={"ID":"5d83608f-bcb0-4aab-b14e-29846561a3d3","Type":"ContainerDied","Data":"eec07909c0ab2a065e177d21c7cae12a71d37a6aba1c35630a172459f2a204d8"} Oct 30 22:03:21.630163 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:21.630155 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw" event={"ID":"5d83608f-bcb0-4aab-b14e-29846561a3d3","Type":"ContainerDied","Data":"4e68d60430e1cf08f68b979f6ad3f14b97e062866971fc892c9a3d296cc4c7a8"} Oct 30 22:03:21.630267 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:21.630182 4560 scope.go:122] "RemoveContainer" containerID="eec07909c0ab2a065e177d21c7cae12a71d37a6aba1c35630a172459f2a204d8" Oct 30 22:03:21.642984 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:21.642970 4560 scope.go:122] "RemoveContainer" containerID="eec07909c0ab2a065e177d21c7cae12a71d37a6aba1c35630a172459f2a204d8" Oct 30 22:03:21.643240 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:03:21.643224 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eec07909c0ab2a065e177d21c7cae12a71d37a6aba1c35630a172459f2a204d8\": container with ID starting with eec07909c0ab2a065e177d21c7cae12a71d37a6aba1c35630a172459f2a204d8 not found: ID does not exist" containerID="eec07909c0ab2a065e177d21c7cae12a71d37a6aba1c35630a172459f2a204d8" Oct 30 22:03:21.643292 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:21.643249 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eec07909c0ab2a065e177d21c7cae12a71d37a6aba1c35630a172459f2a204d8"} err="failed to get container status \"eec07909c0ab2a065e177d21c7cae12a71d37a6aba1c35630a172459f2a204d8\": rpc error: code = NotFound desc = could not find container \"eec07909c0ab2a065e177d21c7cae12a71d37a6aba1c35630a172459f2a204d8\": container with ID starting with eec07909c0ab2a065e177d21c7cae12a71d37a6aba1c35630a172459f2a204d8 not found: ID does not exist" Oct 30 22:03:21.721861 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:21.721832 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d83608f-bcb0-4aab-b14e-29846561a3d3-istio-token\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:21.721861 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:21.721860 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d83608f-bcb0-4aab-b14e-29846561a3d3-istio-envoy\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:21.722021 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:21.721874 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkpsq\" (UniqueName: \"kubernetes.io/projected/5d83608f-bcb0-4aab-b14e-29846561a3d3-kube-api-access-nkpsq\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:21.722021 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:21.721888 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d83608f-bcb0-4aab-b14e-29846561a3d3-istio-data\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:21.722021 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:21.721900 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d83608f-bcb0-4aab-b14e-29846561a3d3-istio-podinfo\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:21.859255 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:21.859209 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw"] Oct 30 22:03:21.861643 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:21.861618 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-991335-openshift-default-6c6b8d69bc-zrttw"] Oct 30 22:03:22.575466 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:22.575433 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d83608f-bcb0-4aab-b14e-29846561a3d3" path="/var/lib/kubelet/pods/5d83608f-bcb0-4aab-b14e-29846561a3d3/volumes" Oct 30 22:03:32.068196 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:32.068165 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:03:47.477507 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:47.477474 4560 scope.go:122] "RemoveContainer" containerID="d0b9739a46201a3c194276e83be7def8ac065a7c348f7f603e9c4b8e794b1b45" Oct 30 22:03:47.486403 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:47.486381 4560 scope.go:122] "RemoveContainer" containerID="50babdb387fe22d83b206da377e53fe5e8d81ccf75ea603fdca5ab79e0322fb2" Oct 30 22:03:47.494694 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:47.494673 4560 scope.go:122] "RemoveContainer" containerID="eb6eebf28355400d6615c9a7a1b8fb685e47af620c84f51016c2fa4a8b686048" Oct 30 22:03:49.538470 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:49.538426 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ac3eeab-lp8mg"] Oct 30 22:03:49.538980 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:49.538843 4560 cpu_manager.go:414] "containerMap: removing container" podUID="80066c33-43b8-4393-af50-10b6a87e84f3" containerName="revoke-keys" Oct 30 22:03:49.538980 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:49.538857 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80066c33-43b8-4393-af50-10b6a87e84f3" containerName="revoke-keys" Oct 30 22:03:49.538980 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:49.538875 4560 cpu_manager.go:414] "containerMap: removing container" podUID="5d83608f-bcb0-4aab-b14e-29846561a3d3" containerName="istio-proxy" Oct 30 22:03:49.538980 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:49.538880 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d83608f-bcb0-4aab-b14e-29846561a3d3" containerName="istio-proxy" Oct 30 22:03:49.538980 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:49.538941 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="80066c33-43b8-4393-af50-10b6a87e84f3" containerName="revoke-keys" Oct 30 22:03:49.538980 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:49.538951 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d83608f-bcb0-4aab-b14e-29846561a3d3" containerName="istio-proxy" Oct 30 22:03:49.543473 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:49.543457 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac3eeab-lp8mg" Oct 30 22:03:49.640331 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:49.640297 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:03:49.640538 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:49.640369 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:03:49.640538 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:49.640402 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:03:49.640538 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:49.640400 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:03:49.642881 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:49.642852 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ac3eeab-lp8mg"] Oct 30 22:03:49.683235 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:03:49.683204 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:03:49.716009 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:49.715977 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjg5w\" (UniqueName: \"kubernetes.io/projected/81d18e45-cbee-4f35-97df-8f80472ba89a-kube-api-access-cjg5w\") pod \"maas-api-revoke-sub-7ac3eeab-lp8mg\" (UID: \"81d18e45-cbee-4f35-97df-8f80472ba89a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac3eeab-lp8mg" Oct 30 22:03:49.716181 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:49.716016 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81d18e45-cbee-4f35-97df-8f80472ba89a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ac3eeab-lp8mg\" (UID: \"81d18e45-cbee-4f35-97df-8f80472ba89a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac3eeab-lp8mg" Oct 30 22:03:49.716246 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:49.716228 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81d18e45-cbee-4f35-97df-8f80472ba89a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ac3eeab-lp8mg\" (UID: \"81d18e45-cbee-4f35-97df-8f80472ba89a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac3eeab-lp8mg" Oct 30 22:03:49.817171 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:49.817081 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjg5w\" (UniqueName: \"kubernetes.io/projected/81d18e45-cbee-4f35-97df-8f80472ba89a-kube-api-access-cjg5w\") pod \"maas-api-revoke-sub-7ac3eeab-lp8mg\" (UID: \"81d18e45-cbee-4f35-97df-8f80472ba89a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac3eeab-lp8mg" Oct 30 22:03:49.817171 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:49.817120 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81d18e45-cbee-4f35-97df-8f80472ba89a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ac3eeab-lp8mg\" (UID: \"81d18e45-cbee-4f35-97df-8f80472ba89a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac3eeab-lp8mg" Oct 30 22:03:49.817450 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:49.817213 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81d18e45-cbee-4f35-97df-8f80472ba89a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ac3eeab-lp8mg\" (UID: \"81d18e45-cbee-4f35-97df-8f80472ba89a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac3eeab-lp8mg" Oct 30 22:03:49.817894 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:49.817874 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81d18e45-cbee-4f35-97df-8f80472ba89a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ac3eeab-lp8mg\" (UID: \"81d18e45-cbee-4f35-97df-8f80472ba89a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac3eeab-lp8mg" Oct 30 22:03:49.819593 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:49.819573 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81d18e45-cbee-4f35-97df-8f80472ba89a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ac3eeab-lp8mg\" (UID: \"81d18e45-cbee-4f35-97df-8f80472ba89a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac3eeab-lp8mg" Oct 30 22:03:49.875177 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:49.875148 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjg5w\" (UniqueName: \"kubernetes.io/projected/81d18e45-cbee-4f35-97df-8f80472ba89a-kube-api-access-cjg5w\") pod \"maas-api-revoke-sub-7ac3eeab-lp8mg\" (UID: \"81d18e45-cbee-4f35-97df-8f80472ba89a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac3eeab-lp8mg" Oct 30 22:03:50.156391 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:50.156293 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac3eeab-lp8mg" Oct 30 22:03:50.528328 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:50.528295 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ac3eeab-lp8mg"] Oct 30 22:03:50.533905 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:03:50.533877 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81d18e45_cbee_4f35_97df_8f80472ba89a.slice/crio-452c4709e1bcd6417b4ec609c0ac65eaaaacdb0b762359c91dded9db4edc4cc3 WatchSource:0}: Error finding container 452c4709e1bcd6417b4ec609c0ac65eaaaacdb0b762359c91dded9db4edc4cc3: Status 404 returned error can't find the container with id 452c4709e1bcd6417b4ec609c0ac65eaaaacdb0b762359c91dded9db4edc4cc3 Oct 30 22:03:50.794810 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:50.794730 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac3eeab-lp8mg" event={"ID":"81d18e45-cbee-4f35-97df-8f80472ba89a","Type":"ContainerStarted","Data":"452c4709e1bcd6417b4ec609c0ac65eaaaacdb0b762359c91dded9db4edc4cc3"} Oct 30 22:03:51.802079 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:51.802046 4560 generic.go:356] "Generic (PLEG): container finished" podID="81d18e45-cbee-4f35-97df-8f80472ba89a" containerID="0b0591f02672f6c4924d3518a1baf880fc85281f53c3fc2778b6d60157756e6b" exitCode=0 Oct 30 22:03:51.802501 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:51.802095 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac3eeab-lp8mg" event={"ID":"81d18e45-cbee-4f35-97df-8f80472ba89a","Type":"ContainerDied","Data":"0b0591f02672f6c4924d3518a1baf880fc85281f53c3fc2778b6d60157756e6b"} Oct 30 22:03:52.955953 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:52.955922 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac3eeab-lp8mg" Oct 30 22:03:53.057634 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:53.057595 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81d18e45-cbee-4f35-97df-8f80472ba89a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81d18e45-cbee-4f35-97df-8f80472ba89a-maas-api-xks-ca\") pod \"81d18e45-cbee-4f35-97df-8f80472ba89a\" (UID: \"81d18e45-cbee-4f35-97df-8f80472ba89a\") " Oct 30 22:03:53.057817 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:53.057667 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81d18e45-cbee-4f35-97df-8f80472ba89a-kube-api-access-cjg5w\" (UniqueName: \"kubernetes.io/projected/81d18e45-cbee-4f35-97df-8f80472ba89a-kube-api-access-cjg5w\") pod \"81d18e45-cbee-4f35-97df-8f80472ba89a\" (UID: \"81d18e45-cbee-4f35-97df-8f80472ba89a\") " Oct 30 22:03:53.057817 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:53.057695 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81d18e45-cbee-4f35-97df-8f80472ba89a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81d18e45-cbee-4f35-97df-8f80472ba89a-maas-api-openshift-ca\") pod \"81d18e45-cbee-4f35-97df-8f80472ba89a\" (UID: \"81d18e45-cbee-4f35-97df-8f80472ba89a\") " Oct 30 22:03:53.058138 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:53.058117 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81d18e45-cbee-4f35-97df-8f80472ba89a-maas-api-openshift-ca" pod "81d18e45-cbee-4f35-97df-8f80472ba89a" (UID: "81d18e45-cbee-4f35-97df-8f80472ba89a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:53.059757 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:53.059701 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81d18e45-cbee-4f35-97df-8f80472ba89a-maas-api-xks-ca" pod "81d18e45-cbee-4f35-97df-8f80472ba89a" (UID: "81d18e45-cbee-4f35-97df-8f80472ba89a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:53.059853 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:53.059827 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81d18e45-cbee-4f35-97df-8f80472ba89a-kube-api-access-cjg5w" pod "81d18e45-cbee-4f35-97df-8f80472ba89a" (UID: "81d18e45-cbee-4f35-97df-8f80472ba89a"). InnerVolumeSpecName "kube-api-access-cjg5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:53.159436 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:53.159400 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81d18e45-cbee-4f35-97df-8f80472ba89a-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:53.159436 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:53.159431 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81d18e45-cbee-4f35-97df-8f80472ba89a-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:53.159436 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:53.159441 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjg5w\" (UniqueName: \"kubernetes.io/projected/81d18e45-cbee-4f35-97df-8f80472ba89a-kube-api-access-cjg5w\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:53.814622 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:53.814593 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac3eeab-lp8mg" Oct 30 22:03:53.814793 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:53.814623 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac3eeab-lp8mg" event={"ID":"81d18e45-cbee-4f35-97df-8f80472ba89a","Type":"ContainerDied","Data":"452c4709e1bcd6417b4ec609c0ac65eaaaacdb0b762359c91dded9db4edc4cc3"} Oct 30 22:03:53.814793 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:53.814673 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="452c4709e1bcd6417b4ec609c0ac65eaaaacdb0b762359c91dded9db4edc4cc3" Oct 30 22:03:57.161946 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:57.161864 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-28c192-q5tc2"] Oct 30 22:03:57.208413 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:57.208351 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-28c192-q5tc2"] Oct 30 22:03:58.574884 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:03:58.574848 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80066c33-43b8-4393-af50-10b6a87e84f3" path="/var/lib/kubelet/pods/80066c33-43b8-4393-af50-10b6a87e84f3/volumes" Oct 30 22:04:04.466600 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:04.466562 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:04:04.618635 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:04.618599 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9"] Oct 30 22:04:04.618896 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:04.618855 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" podUID="7f4405ac-93f4-4a1d-a335-5990238c4294" containerName="istio-proxy" containerID="cri-o://feb158cd0379b553dd6379400a6696ab4eaa69c5cc34a49485737fbb7af15554" gracePeriod=30 Oct 30 22:04:05.384362 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:05.383394 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg"] Oct 30 22:04:05.389699 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:05.386322 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc"] Oct 30 22:04:05.416382 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:05.416171 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87"] Oct 30 22:04:05.624419 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:05.624378 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-87eb6a-gc5xh"] Oct 30 22:04:05.624961 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:05.624933 4560 cpu_manager.go:414] "containerMap: removing container" podUID="81d18e45-cbee-4f35-97df-8f80472ba89a" containerName="revoke-keys" Oct 30 22:04:05.624961 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:05.624957 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81d18e45-cbee-4f35-97df-8f80472ba89a" containerName="revoke-keys" Oct 30 22:04:05.625127 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:05.625051 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="81d18e45-cbee-4f35-97df-8f80472ba89a" containerName="revoke-keys" Oct 30 22:04:05.636667 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:05.636597 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-87eb6a-gc5xh" Oct 30 22:04:05.697365 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:05.685396 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-87eb6a-gc5xh"] Oct 30 22:04:05.704093 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:05.704055 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:04:05.704446 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:05.704425 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:04:05.704630 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:05.704616 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:04:05.704838 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:05.704821 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:04:05.711517 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:05.711497 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e61932e-01e1-4c1d-893e-632817e91104-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-87eb6a-gc5xh\" (UID: \"3e61932e-01e1-4c1d-893e-632817e91104\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-87eb6a-gc5xh" Oct 30 22:04:05.711720 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:05.711700 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpvvh\" (UniqueName: \"kubernetes.io/projected/3e61932e-01e1-4c1d-893e-632817e91104-kube-api-access-tpvvh\") pod \"maas-api-revoke-keys-e2e-api-b-87eb6a-gc5xh\" (UID: \"3e61932e-01e1-4c1d-893e-632817e91104\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-87eb6a-gc5xh" Oct 30 22:04:05.712464 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:05.712440 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e61932e-01e1-4c1d-893e-632817e91104-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-87eb6a-gc5xh\" (UID: \"3e61932e-01e1-4c1d-893e-632817e91104\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-87eb6a-gc5xh" Oct 30 22:04:05.813457 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:05.813421 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpvvh\" (UniqueName: \"kubernetes.io/projected/3e61932e-01e1-4c1d-893e-632817e91104-kube-api-access-tpvvh\") pod \"maas-api-revoke-keys-e2e-api-b-87eb6a-gc5xh\" (UID: \"3e61932e-01e1-4c1d-893e-632817e91104\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-87eb6a-gc5xh" Oct 30 22:04:05.813736 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:05.813711 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e61932e-01e1-4c1d-893e-632817e91104-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-87eb6a-gc5xh\" (UID: \"3e61932e-01e1-4c1d-893e-632817e91104\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-87eb6a-gc5xh" Oct 30 22:04:05.813929 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:05.813911 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e61932e-01e1-4c1d-893e-632817e91104-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-87eb6a-gc5xh\" (UID: \"3e61932e-01e1-4c1d-893e-632817e91104\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-87eb6a-gc5xh" Oct 30 22:04:05.814547 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:05.814528 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e61932e-01e1-4c1d-893e-632817e91104-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-87eb6a-gc5xh\" (UID: \"3e61932e-01e1-4c1d-893e-632817e91104\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-87eb6a-gc5xh" Oct 30 22:04:05.817013 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:05.816993 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e61932e-01e1-4c1d-893e-632817e91104-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-87eb6a-gc5xh\" (UID: \"3e61932e-01e1-4c1d-893e-632817e91104\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-87eb6a-gc5xh" Oct 30 22:04:05.913222 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:05.913144 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpvvh\" (UniqueName: \"kubernetes.io/projected/3e61932e-01e1-4c1d-893e-632817e91104-kube-api-access-tpvvh\") pod \"maas-api-revoke-keys-e2e-api-b-87eb6a-gc5xh\" (UID: \"3e61932e-01e1-4c1d-893e-632817e91104\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-87eb6a-gc5xh" Oct 30 22:04:05.994647 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:05.994616 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-87eb6a-gc5xh" Oct 30 22:04:06.109371 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:06.109315 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ed76b8-7fnl9"] Oct 30 22:04:06.114006 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:06.113984 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ed76b8-7fnl9" Oct 30 22:04:06.242909 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:06.242415 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/405964a0-6ee3-4be8-8bac-0baaee68c93b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ed76b8-7fnl9\" (UID: \"405964a0-6ee3-4be8-8bac-0baaee68c93b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ed76b8-7fnl9" Oct 30 22:04:06.242909 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:06.242475 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9vrq\" (UniqueName: \"kubernetes.io/projected/405964a0-6ee3-4be8-8bac-0baaee68c93b-kube-api-access-b9vrq\") pod \"maas-api-revoke-keys-e2e-worker-w2-ed76b8-7fnl9\" (UID: \"405964a0-6ee3-4be8-8bac-0baaee68c93b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ed76b8-7fnl9" Oct 30 22:04:06.242909 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:06.242677 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/405964a0-6ee3-4be8-8bac-0baaee68c93b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ed76b8-7fnl9\" (UID: \"405964a0-6ee3-4be8-8bac-0baaee68c93b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ed76b8-7fnl9" Oct 30 22:04:06.263371 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:06.260685 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ed76b8-7fnl9"] Oct 30 22:04:06.356368 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:06.344949 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/405964a0-6ee3-4be8-8bac-0baaee68c93b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ed76b8-7fnl9\" (UID: \"405964a0-6ee3-4be8-8bac-0baaee68c93b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ed76b8-7fnl9" Oct 30 22:04:06.356368 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:06.345025 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/405964a0-6ee3-4be8-8bac-0baaee68c93b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ed76b8-7fnl9\" (UID: \"405964a0-6ee3-4be8-8bac-0baaee68c93b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ed76b8-7fnl9" Oct 30 22:04:06.356368 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:06.345070 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9vrq\" (UniqueName: \"kubernetes.io/projected/405964a0-6ee3-4be8-8bac-0baaee68c93b-kube-api-access-b9vrq\") pod \"maas-api-revoke-keys-e2e-worker-w2-ed76b8-7fnl9\" (UID: \"405964a0-6ee3-4be8-8bac-0baaee68c93b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ed76b8-7fnl9" Oct 30 22:04:06.375724 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:06.375583 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/405964a0-6ee3-4be8-8bac-0baaee68c93b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ed76b8-7fnl9\" (UID: \"405964a0-6ee3-4be8-8bac-0baaee68c93b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ed76b8-7fnl9" Oct 30 22:04:06.410822 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:06.408441 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/405964a0-6ee3-4be8-8bac-0baaee68c93b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ed76b8-7fnl9\" (UID: \"405964a0-6ee3-4be8-8bac-0baaee68c93b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ed76b8-7fnl9" Oct 30 22:04:06.417259 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:06.417236 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9vrq\" (UniqueName: \"kubernetes.io/projected/405964a0-6ee3-4be8-8bac-0baaee68c93b-kube-api-access-b9vrq\") pod \"maas-api-revoke-keys-e2e-worker-w2-ed76b8-7fnl9\" (UID: \"405964a0-6ee3-4be8-8bac-0baaee68c93b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ed76b8-7fnl9" Oct 30 22:04:06.477363 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:06.475569 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ed76b8-7fnl9" Oct 30 22:04:06.741578 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:06.740099 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-87eb6a-gc5xh"] Oct 30 22:04:06.745429 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:04:06.745391 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e61932e_01e1_4c1d_893e_632817e91104.slice/crio-75d0e01ba9f20016b0a15aa68461d3291fd7948c805a2818d327b38dac4d8447 WatchSource:0}: Error finding container 75d0e01ba9f20016b0a15aa68461d3291fd7948c805a2818d327b38dac4d8447: Status 404 returned error can't find the container with id 75d0e01ba9f20016b0a15aa68461d3291fd7948c805a2818d327b38dac4d8447 Oct 30 22:04:06.920227 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:06.920197 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-87eb6a-gc5xh" event={"ID":"3e61932e-01e1-4c1d-893e-632817e91104","Type":"ContainerStarted","Data":"75d0e01ba9f20016b0a15aa68461d3291fd7948c805a2818d327b38dac4d8447"} Oct 30 22:04:07.019697 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:07.019622 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ed76b8-7fnl9"] Oct 30 22:04:07.025517 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:04:07.025482 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405964a0_6ee3_4be8_8bac_0baaee68c93b.slice/crio-5c84d6413e9dbf41c06184cc49ba9c93fe5634e1075ac2fbe395a1fdff78e250 WatchSource:0}: Error finding container 5c84d6413e9dbf41c06184cc49ba9c93fe5634e1075ac2fbe395a1fdff78e250: Status 404 returned error can't find the container with id 5c84d6413e9dbf41c06184cc49ba9c93fe5634e1075ac2fbe395a1fdff78e250 Oct 30 22:04:07.927083 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:07.927047 4560 generic.go:356] "Generic (PLEG): container finished" podID="3e61932e-01e1-4c1d-893e-632817e91104" containerID="16ef64a6710faa33523f9bff454e6bc16069579027fcf1a9b1c8988a7b729300" exitCode=0 Oct 30 22:04:07.927591 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:07.927132 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-87eb6a-gc5xh" event={"ID":"3e61932e-01e1-4c1d-893e-632817e91104","Type":"ContainerDied","Data":"16ef64a6710faa33523f9bff454e6bc16069579027fcf1a9b1c8988a7b729300"} Oct 30 22:04:07.928916 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:07.928888 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ed76b8-7fnl9" event={"ID":"405964a0-6ee3-4be8-8bac-0baaee68c93b","Type":"ContainerStarted","Data":"ca997a0b05e4a7beb50e14fd8fb1dbb7e999f0ec6445261ca9aaf06fc6615c1e"} Oct 30 22:04:07.929070 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:07.928937 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ed76b8-7fnl9" event={"ID":"405964a0-6ee3-4be8-8bac-0baaee68c93b","Type":"ContainerStarted","Data":"5c84d6413e9dbf41c06184cc49ba9c93fe5634e1075ac2fbe395a1fdff78e250"} Oct 30 22:04:08.146115 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:08.146053 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ed76b8-7fnl9" podStartSLOduration=2.146039193 podStartE2EDuration="2.146039193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:08.145124453 +0000 UTC m=+3023.699824229" watchObservedRunningTime="2026-10-30 22:04:08.146039193 +0000 UTC m=+3023.700738976" Oct 30 22:04:08.935687 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:08.935656 4560 generic.go:356] "Generic (PLEG): container finished" podID="405964a0-6ee3-4be8-8bac-0baaee68c93b" containerID="ca997a0b05e4a7beb50e14fd8fb1dbb7e999f0ec6445261ca9aaf06fc6615c1e" exitCode=0 Oct 30 22:04:08.936149 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:08.935736 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ed76b8-7fnl9" event={"ID":"405964a0-6ee3-4be8-8bac-0baaee68c93b","Type":"ContainerDied","Data":"ca997a0b05e4a7beb50e14fd8fb1dbb7e999f0ec6445261ca9aaf06fc6615c1e"} Oct 30 22:04:09.083231 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.083201 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-87eb6a-gc5xh" Oct 30 22:04:09.235828 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.235798 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e61932e-01e1-4c1d-893e-632817e91104-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e61932e-01e1-4c1d-893e-632817e91104-maas-api-xks-ca\") pod \"3e61932e-01e1-4c1d-893e-632817e91104\" (UID: \"3e61932e-01e1-4c1d-893e-632817e91104\") " Oct 30 22:04:09.236002 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.235838 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e61932e-01e1-4c1d-893e-632817e91104-kube-api-access-tpvvh\" (UniqueName: \"kubernetes.io/projected/3e61932e-01e1-4c1d-893e-632817e91104-kube-api-access-tpvvh\") pod \"3e61932e-01e1-4c1d-893e-632817e91104\" (UID: \"3e61932e-01e1-4c1d-893e-632817e91104\") " Oct 30 22:04:09.236002 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.235929 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e61932e-01e1-4c1d-893e-632817e91104-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e61932e-01e1-4c1d-893e-632817e91104-maas-api-openshift-ca\") pod \"3e61932e-01e1-4c1d-893e-632817e91104\" (UID: \"3e61932e-01e1-4c1d-893e-632817e91104\") " Oct 30 22:04:09.236320 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.236295 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e61932e-01e1-4c1d-893e-632817e91104-maas-api-openshift-ca" pod "3e61932e-01e1-4c1d-893e-632817e91104" (UID: "3e61932e-01e1-4c1d-893e-632817e91104"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:09.236457 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.236440 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e61932e-01e1-4c1d-893e-632817e91104-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:09.237985 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.237963 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e61932e-01e1-4c1d-893e-632817e91104-kube-api-access-tpvvh" pod "3e61932e-01e1-4c1d-893e-632817e91104" (UID: "3e61932e-01e1-4c1d-893e-632817e91104"). InnerVolumeSpecName "kube-api-access-tpvvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:09.238143 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.237981 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e61932e-01e1-4c1d-893e-632817e91104-maas-api-xks-ca" pod "3e61932e-01e1-4c1d-893e-632817e91104" (UID: "3e61932e-01e1-4c1d-893e-632817e91104"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:09.337191 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.337153 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e61932e-01e1-4c1d-893e-632817e91104-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:09.337191 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.337186 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpvvh\" (UniqueName: \"kubernetes.io/projected/3e61932e-01e1-4c1d-893e-632817e91104-kube-api-access-tpvvh\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:09.847923 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.847896 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" Oct 30 22:04:09.943663 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.943572 4560 generic.go:356] "Generic (PLEG): container finished" podID="7f4405ac-93f4-4a1d-a335-5990238c4294" containerID="feb158cd0379b553dd6379400a6696ab4eaa69c5cc34a49485737fbb7af15554" exitCode=0 Oct 30 22:04:09.943663 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.943630 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" Oct 30 22:04:09.944228 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.943659 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" event={"ID":"7f4405ac-93f4-4a1d-a335-5990238c4294","Type":"ContainerDied","Data":"feb158cd0379b553dd6379400a6696ab4eaa69c5cc34a49485737fbb7af15554"} Oct 30 22:04:09.944228 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.943727 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9" event={"ID":"7f4405ac-93f4-4a1d-a335-5990238c4294","Type":"ContainerDied","Data":"cb20ac229596075cb5e8ee9ad4ca8d42fc90f6c4b723073a435e179e830959e2"} Oct 30 22:04:09.944228 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.943762 4560 scope.go:122] "RemoveContainer" containerID="feb158cd0379b553dd6379400a6696ab4eaa69c5cc34a49485737fbb7af15554" Oct 30 22:04:09.944228 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.943988 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f4405ac-93f4-4a1d-a335-5990238c4294-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f4405ac-93f4-4a1d-a335-5990238c4294-workload-certs\") pod \"7f4405ac-93f4-4a1d-a335-5990238c4294\" (UID: \"7f4405ac-93f4-4a1d-a335-5990238c4294\") " Oct 30 22:04:09.944228 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.944047 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f4405ac-93f4-4a1d-a335-5990238c4294-istio-token\" (UniqueName: \"kubernetes.io/projected/7f4405ac-93f4-4a1d-a335-5990238c4294-istio-token\") pod \"7f4405ac-93f4-4a1d-a335-5990238c4294\" (UID: \"7f4405ac-93f4-4a1d-a335-5990238c4294\") " Oct 30 22:04:09.944228 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.944072 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f4405ac-93f4-4a1d-a335-5990238c4294-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f4405ac-93f4-4a1d-a335-5990238c4294-istio-podinfo\") pod \"7f4405ac-93f4-4a1d-a335-5990238c4294\" (UID: \"7f4405ac-93f4-4a1d-a335-5990238c4294\") " Oct 30 22:04:09.944228 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.944154 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f4405ac-93f4-4a1d-a335-5990238c4294-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4405ac-93f4-4a1d-a335-5990238c4294-credential-socket\") pod \"7f4405ac-93f4-4a1d-a335-5990238c4294\" (UID: \"7f4405ac-93f4-4a1d-a335-5990238c4294\") " Oct 30 22:04:09.944228 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.944212 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f4405ac-93f4-4a1d-a335-5990238c4294-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f4405ac-93f4-4a1d-a335-5990238c4294-istio-data\") pod \"7f4405ac-93f4-4a1d-a335-5990238c4294\" (UID: \"7f4405ac-93f4-4a1d-a335-5990238c4294\") " Oct 30 22:04:09.944684 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.944240 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f4405ac-93f4-4a1d-a335-5990238c4294-kube-api-access-7jvln\" (UniqueName: \"kubernetes.io/projected/7f4405ac-93f4-4a1d-a335-5990238c4294-kube-api-access-7jvln\") pod \"7f4405ac-93f4-4a1d-a335-5990238c4294\" (UID: \"7f4405ac-93f4-4a1d-a335-5990238c4294\") " Oct 30 22:04:09.944684 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.944281 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f4405ac-93f4-4a1d-a335-5990238c4294-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f4405ac-93f4-4a1d-a335-5990238c4294-istiod-ca-cert\") pod \"7f4405ac-93f4-4a1d-a335-5990238c4294\" (UID: \"7f4405ac-93f4-4a1d-a335-5990238c4294\") " Oct 30 22:04:09.944684 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.944333 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f4405ac-93f4-4a1d-a335-5990238c4294-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f4405ac-93f4-4a1d-a335-5990238c4294-istio-envoy\") pod \"7f4405ac-93f4-4a1d-a335-5990238c4294\" (UID: \"7f4405ac-93f4-4a1d-a335-5990238c4294\") " Oct 30 22:04:09.944684 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.944382 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f4405ac-93f4-4a1d-a335-5990238c4294-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4405ac-93f4-4a1d-a335-5990238c4294-workload-socket\") pod \"7f4405ac-93f4-4a1d-a335-5990238c4294\" (UID: \"7f4405ac-93f4-4a1d-a335-5990238c4294\") " Oct 30 22:04:09.944684 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.944637 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f4405ac-93f4-4a1d-a335-5990238c4294-workload-certs" pod "7f4405ac-93f4-4a1d-a335-5990238c4294" (UID: "7f4405ac-93f4-4a1d-a335-5990238c4294"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:09.944939 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.944810 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f4405ac-93f4-4a1d-a335-5990238c4294-credential-socket" pod "7f4405ac-93f4-4a1d-a335-5990238c4294" (UID: "7f4405ac-93f4-4a1d-a335-5990238c4294"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:09.944939 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.944833 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f4405ac-93f4-4a1d-a335-5990238c4294-workload-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:09.944939 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.944847 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f4405ac-93f4-4a1d-a335-5990238c4294-workload-socket" pod "7f4405ac-93f4-4a1d-a335-5990238c4294" (UID: "7f4405ac-93f4-4a1d-a335-5990238c4294"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:09.944939 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.944896 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f4405ac-93f4-4a1d-a335-5990238c4294-istiod-ca-cert" pod "7f4405ac-93f4-4a1d-a335-5990238c4294" (UID: "7f4405ac-93f4-4a1d-a335-5990238c4294"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:09.944939 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.944861 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f4405ac-93f4-4a1d-a335-5990238c4294-istio-data" pod "7f4405ac-93f4-4a1d-a335-5990238c4294" (UID: "7f4405ac-93f4-4a1d-a335-5990238c4294"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:09.945891 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.945870 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-87eb6a-gc5xh" event={"ID":"3e61932e-01e1-4c1d-893e-632817e91104","Type":"ContainerDied","Data":"75d0e01ba9f20016b0a15aa68461d3291fd7948c805a2818d327b38dac4d8447"} Oct 30 22:04:09.946060 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.945914 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75d0e01ba9f20016b0a15aa68461d3291fd7948c805a2818d327b38dac4d8447" Oct 30 22:04:09.946060 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.945948 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-87eb6a-gc5xh" Oct 30 22:04:09.947391 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.946820 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f4405ac-93f4-4a1d-a335-5990238c4294-istio-token" pod "7f4405ac-93f4-4a1d-a335-5990238c4294" (UID: "7f4405ac-93f4-4a1d-a335-5990238c4294"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:09.947391 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.947064 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f4405ac-93f4-4a1d-a335-5990238c4294-istio-podinfo" pod "7f4405ac-93f4-4a1d-a335-5990238c4294" (UID: "7f4405ac-93f4-4a1d-a335-5990238c4294"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:04:09.947391 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.947334 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f4405ac-93f4-4a1d-a335-5990238c4294-kube-api-access-7jvln" pod "7f4405ac-93f4-4a1d-a335-5990238c4294" (UID: "7f4405ac-93f4-4a1d-a335-5990238c4294"). InnerVolumeSpecName "kube-api-access-7jvln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:09.947581 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.947436 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f4405ac-93f4-4a1d-a335-5990238c4294-istio-envoy" pod "7f4405ac-93f4-4a1d-a335-5990238c4294" (UID: "7f4405ac-93f4-4a1d-a335-5990238c4294"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:09.956445 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.956034 4560 scope.go:122] "RemoveContainer" containerID="de6e083df99964d0fec1de18f4a318649336521e9d7ad8b20d61d0c3d1dc1c10" Oct 30 22:04:09.968010 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.967983 4560 scope.go:122] "RemoveContainer" containerID="feb158cd0379b553dd6379400a6696ab4eaa69c5cc34a49485737fbb7af15554" Oct 30 22:04:09.968400 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:04:09.968375 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feb158cd0379b553dd6379400a6696ab4eaa69c5cc34a49485737fbb7af15554\": container with ID starting with feb158cd0379b553dd6379400a6696ab4eaa69c5cc34a49485737fbb7af15554 not found: ID does not exist" containerID="feb158cd0379b553dd6379400a6696ab4eaa69c5cc34a49485737fbb7af15554" Oct 30 22:04:09.968488 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.968413 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feb158cd0379b553dd6379400a6696ab4eaa69c5cc34a49485737fbb7af15554"} err="failed to get container status \"feb158cd0379b553dd6379400a6696ab4eaa69c5cc34a49485737fbb7af15554\": rpc error: code = NotFound desc = could not find container \"feb158cd0379b553dd6379400a6696ab4eaa69c5cc34a49485737fbb7af15554\": container with ID starting with feb158cd0379b553dd6379400a6696ab4eaa69c5cc34a49485737fbb7af15554 not found: ID does not exist" Oct 30 22:04:09.968488 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.968438 4560 scope.go:122] "RemoveContainer" containerID="de6e083df99964d0fec1de18f4a318649336521e9d7ad8b20d61d0c3d1dc1c10" Oct 30 22:04:09.968770 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:04:09.968746 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de6e083df99964d0fec1de18f4a318649336521e9d7ad8b20d61d0c3d1dc1c10\": container with ID starting with de6e083df99964d0fec1de18f4a318649336521e9d7ad8b20d61d0c3d1dc1c10 not found: ID does not exist" containerID="de6e083df99964d0fec1de18f4a318649336521e9d7ad8b20d61d0c3d1dc1c10" Oct 30 22:04:09.968864 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:09.968773 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de6e083df99964d0fec1de18f4a318649336521e9d7ad8b20d61d0c3d1dc1c10"} err="failed to get container status \"de6e083df99964d0fec1de18f4a318649336521e9d7ad8b20d61d0c3d1dc1c10\": rpc error: code = NotFound desc = could not find container \"de6e083df99964d0fec1de18f4a318649336521e9d7ad8b20d61d0c3d1dc1c10\": container with ID starting with de6e083df99964d0fec1de18f4a318649336521e9d7ad8b20d61d0c3d1dc1c10 not found: ID does not exist" Oct 30 22:04:10.045924 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:10.045885 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f4405ac-93f4-4a1d-a335-5990238c4294-istio-token\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:10.045924 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:10.045923 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f4405ac-93f4-4a1d-a335-5990238c4294-istio-podinfo\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:10.046100 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:10.045941 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4405ac-93f4-4a1d-a335-5990238c4294-credential-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:10.046100 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:10.045955 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f4405ac-93f4-4a1d-a335-5990238c4294-istio-data\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:10.046100 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:10.045973 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jvln\" (UniqueName: \"kubernetes.io/projected/7f4405ac-93f4-4a1d-a335-5990238c4294-kube-api-access-7jvln\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:10.046100 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:10.045988 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f4405ac-93f4-4a1d-a335-5990238c4294-istiod-ca-cert\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:10.046100 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:10.046000 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f4405ac-93f4-4a1d-a335-5990238c4294-istio-envoy\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:10.046100 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:10.046015 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4405ac-93f4-4a1d-a335-5990238c4294-workload-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:10.079291 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:10.079264 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ed76b8-7fnl9" Oct 30 22:04:10.146358 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:10.146312 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/405964a0-6ee3-4be8-8bac-0baaee68c93b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/405964a0-6ee3-4be8-8bac-0baaee68c93b-maas-api-xks-ca\") pod \"405964a0-6ee3-4be8-8bac-0baaee68c93b\" (UID: \"405964a0-6ee3-4be8-8bac-0baaee68c93b\") " Oct 30 22:04:10.146552 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:10.146387 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/405964a0-6ee3-4be8-8bac-0baaee68c93b-kube-api-access-b9vrq\" (UniqueName: \"kubernetes.io/projected/405964a0-6ee3-4be8-8bac-0baaee68c93b-kube-api-access-b9vrq\") pod \"405964a0-6ee3-4be8-8bac-0baaee68c93b\" (UID: \"405964a0-6ee3-4be8-8bac-0baaee68c93b\") " Oct 30 22:04:10.146552 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:10.146449 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/405964a0-6ee3-4be8-8bac-0baaee68c93b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/405964a0-6ee3-4be8-8bac-0baaee68c93b-maas-api-openshift-ca\") pod \"405964a0-6ee3-4be8-8bac-0baaee68c93b\" (UID: \"405964a0-6ee3-4be8-8bac-0baaee68c93b\") " Oct 30 22:04:10.146845 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:10.146824 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/405964a0-6ee3-4be8-8bac-0baaee68c93b-maas-api-openshift-ca" pod "405964a0-6ee3-4be8-8bac-0baaee68c93b" (UID: "405964a0-6ee3-4be8-8bac-0baaee68c93b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:10.148518 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:10.148496 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405964a0-6ee3-4be8-8bac-0baaee68c93b-kube-api-access-b9vrq" pod "405964a0-6ee3-4be8-8bac-0baaee68c93b" (UID: "405964a0-6ee3-4be8-8bac-0baaee68c93b"). InnerVolumeSpecName "kube-api-access-b9vrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:10.148576 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:10.148548 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/405964a0-6ee3-4be8-8bac-0baaee68c93b-maas-api-xks-ca" pod "405964a0-6ee3-4be8-8bac-0baaee68c93b" (UID: "405964a0-6ee3-4be8-8bac-0baaee68c93b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:10.247240 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:10.247204 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/405964a0-6ee3-4be8-8bac-0baaee68c93b-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:10.247240 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:10.247230 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/405964a0-6ee3-4be8-8bac-0baaee68c93b-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:10.247240 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:10.247240 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9vrq\" (UniqueName: \"kubernetes.io/projected/405964a0-6ee3-4be8-8bac-0baaee68c93b-kube-api-access-b9vrq\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:10.358137 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:10.358101 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9"] Oct 30 22:04:10.363719 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:10.363686 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-28c192-openshift-default-5586f598c-r8qq9"] Oct 30 22:04:10.575527 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:10.575452 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f4405ac-93f4-4a1d-a335-5990238c4294" path="/var/lib/kubelet/pods/7f4405ac-93f4-4a1d-a335-5990238c4294/volumes" Oct 30 22:04:10.954370 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:10.954261 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ed76b8-7fnl9" Oct 30 22:04:10.954370 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:10.954283 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ed76b8-7fnl9" event={"ID":"405964a0-6ee3-4be8-8bac-0baaee68c93b","Type":"ContainerDied","Data":"5c84d6413e9dbf41c06184cc49ba9c93fe5634e1075ac2fbe395a1fdff78e250"} Oct 30 22:04:10.954370 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:10.954332 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c84d6413e9dbf41c06184cc49ba9c93fe5634e1075ac2fbe395a1fdff78e250" Oct 30 22:04:19.685117 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:04:19.685075 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:04:20.417658 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.417622 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" podUID="1e044c02-0053-40e8-9f80-e806cd2d9ae5" containerName="main" containerID="cri-o://9dd25fa7195cbf6728318117ab0300f29ff576231a5b8c4958bf02c277efe16a" gracePeriod=45 Oct 30 22:04:20.421097 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.421066 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" podUID="e3281dbe-e8f9-4bab-9ae4-2279e79b7292" containerName="main" containerID="cri-o://44bb8b61144f12209f730c543869497701cc5a3c15dadb47a4a83a781912a01e" gracePeriod=45 Oct 30 22:04:20.442779 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.442750 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" podUID="4dee11ef-9647-47ee-a4b1-686680256542" containerName="main" containerID="cri-o://9b78d5b264887a2ae01f70c4ffe21587c242e470eead7b603c2191bdf76e4c55" gracePeriod=45 Oct 30 22:04:20.716384 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.716232 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" Oct 30 22:04:20.719443 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.719422 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" Oct 30 22:04:20.736083 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.736061 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" Oct 30 22:04:20.860859 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.860829 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e044c02-0053-40e8-9f80-e806cd2d9ae5-home\" (UniqueName: \"kubernetes.io/empty-dir/1e044c02-0053-40e8-9f80-e806cd2d9ae5-home\") pod \"1e044c02-0053-40e8-9f80-e806cd2d9ae5\" (UID: \"1e044c02-0053-40e8-9f80-e806cd2d9ae5\") " Oct 30 22:04:20.861024 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.860892 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-tls-certs\" (UniqueName: \"kubernetes.io/secret/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-tls-certs\") pod \"e3281dbe-e8f9-4bab-9ae4-2279e79b7292\" (UID: \"e3281dbe-e8f9-4bab-9ae4-2279e79b7292\") " Oct 30 22:04:20.861024 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.860917 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e044c02-0053-40e8-9f80-e806cd2d9ae5-kube-api-access-k5t4c\" (UniqueName: \"kubernetes.io/projected/1e044c02-0053-40e8-9f80-e806cd2d9ae5-kube-api-access-k5t4c\") pod \"1e044c02-0053-40e8-9f80-e806cd2d9ae5\" (UID: \"1e044c02-0053-40e8-9f80-e806cd2d9ae5\") " Oct 30 22:04:20.861024 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.860941 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-model-cache\") pod \"e3281dbe-e8f9-4bab-9ae4-2279e79b7292\" (UID: \"e3281dbe-e8f9-4bab-9ae4-2279e79b7292\") " Oct 30 22:04:20.861024 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.860958 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e044c02-0053-40e8-9f80-e806cd2d9ae5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e044c02-0053-40e8-9f80-e806cd2d9ae5-dshm\") pod \"1e044c02-0053-40e8-9f80-e806cd2d9ae5\" (UID: \"1e044c02-0053-40e8-9f80-e806cd2d9ae5\") " Oct 30 22:04:20.861024 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.860988 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dee11ef-9647-47ee-a4b1-686680256542-home\" (UniqueName: \"kubernetes.io/empty-dir/4dee11ef-9647-47ee-a4b1-686680256542-home\") pod \"4dee11ef-9647-47ee-a4b1-686680256542\" (UID: \"4dee11ef-9647-47ee-a4b1-686680256542\") " Oct 30 22:04:20.861323 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.861173 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e044c02-0053-40e8-9f80-e806cd2d9ae5-home" pod "1e044c02-0053-40e8-9f80-e806cd2d9ae5" (UID: "1e044c02-0053-40e8-9f80-e806cd2d9ae5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:20.861323 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.861227 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dee11ef-9647-47ee-a4b1-686680256542-home" pod "4dee11ef-9647-47ee-a4b1-686680256542" (UID: "4dee11ef-9647-47ee-a4b1-686680256542"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:20.861482 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.861429 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-model-cache" pod "e3281dbe-e8f9-4bab-9ae4-2279e79b7292" (UID: "e3281dbe-e8f9-4bab-9ae4-2279e79b7292"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:20.861547 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.861538 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-tmp-dir\") pod \"e3281dbe-e8f9-4bab-9ae4-2279e79b7292\" (UID: \"e3281dbe-e8f9-4bab-9ae4-2279e79b7292\") " Oct 30 22:04:20.861606 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.861581 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dee11ef-9647-47ee-a4b1-686680256542-tls-certs\" (UniqueName: \"kubernetes.io/secret/4dee11ef-9647-47ee-a4b1-686680256542-tls-certs\") pod \"4dee11ef-9647-47ee-a4b1-686680256542\" (UID: \"4dee11ef-9647-47ee-a4b1-686680256542\") " Oct 30 22:04:20.862098 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.861653 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dee11ef-9647-47ee-a4b1-686680256542-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dee11ef-9647-47ee-a4b1-686680256542-tmp-dir\") pod \"4dee11ef-9647-47ee-a4b1-686680256542\" (UID: \"4dee11ef-9647-47ee-a4b1-686680256542\") " Oct 30 22:04:20.862098 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.861707 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dee11ef-9647-47ee-a4b1-686680256542-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dee11ef-9647-47ee-a4b1-686680256542-dshm\") pod \"4dee11ef-9647-47ee-a4b1-686680256542\" (UID: \"4dee11ef-9647-47ee-a4b1-686680256542\") " Oct 30 22:04:20.862098 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.861737 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e044c02-0053-40e8-9f80-e806cd2d9ae5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e044c02-0053-40e8-9f80-e806cd2d9ae5-tmp-dir\") pod \"1e044c02-0053-40e8-9f80-e806cd2d9ae5\" (UID: \"1e044c02-0053-40e8-9f80-e806cd2d9ae5\") " Oct 30 22:04:20.862098 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.861852 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-dshm\") pod \"e3281dbe-e8f9-4bab-9ae4-2279e79b7292\" (UID: \"e3281dbe-e8f9-4bab-9ae4-2279e79b7292\") " Oct 30 22:04:20.862098 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.861882 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dee11ef-9647-47ee-a4b1-686680256542-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dee11ef-9647-47ee-a4b1-686680256542-model-cache\") pod \"4dee11ef-9647-47ee-a4b1-686680256542\" (UID: \"4dee11ef-9647-47ee-a4b1-686680256542\") " Oct 30 22:04:20.862098 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.861920 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-home\" (UniqueName: \"kubernetes.io/empty-dir/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-home\") pod \"e3281dbe-e8f9-4bab-9ae4-2279e79b7292\" (UID: \"e3281dbe-e8f9-4bab-9ae4-2279e79b7292\") " Oct 30 22:04:20.862098 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.861980 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e044c02-0053-40e8-9f80-e806cd2d9ae5-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e044c02-0053-40e8-9f80-e806cd2d9ae5-tls-certs\") pod \"1e044c02-0053-40e8-9f80-e806cd2d9ae5\" (UID: \"1e044c02-0053-40e8-9f80-e806cd2d9ae5\") " Oct 30 22:04:20.862098 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.862020 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-kube-api-access-rfksb\" (UniqueName: \"kubernetes.io/projected/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-kube-api-access-rfksb\") pod \"e3281dbe-e8f9-4bab-9ae4-2279e79b7292\" (UID: \"e3281dbe-e8f9-4bab-9ae4-2279e79b7292\") " Oct 30 22:04:20.862098 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.862036 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dee11ef-9647-47ee-a4b1-686680256542-tmp-dir" pod "4dee11ef-9647-47ee-a4b1-686680256542" (UID: "4dee11ef-9647-47ee-a4b1-686680256542"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:20.862098 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.862061 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dee11ef-9647-47ee-a4b1-686680256542-kube-api-access-6qbtf\" (UniqueName: \"kubernetes.io/projected/4dee11ef-9647-47ee-a4b1-686680256542-kube-api-access-6qbtf\") pod \"4dee11ef-9647-47ee-a4b1-686680256542\" (UID: \"4dee11ef-9647-47ee-a4b1-686680256542\") " Oct 30 22:04:20.863817 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.863181 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-tmp-dir" pod "e3281dbe-e8f9-4bab-9ae4-2279e79b7292" (UID: "e3281dbe-e8f9-4bab-9ae4-2279e79b7292"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:20.863817 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.863474 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e044c02-0053-40e8-9f80-e806cd2d9ae5-tmp-dir" pod "1e044c02-0053-40e8-9f80-e806cd2d9ae5" (UID: "1e044c02-0053-40e8-9f80-e806cd2d9ae5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:20.863817 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.863673 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-home" pod "e3281dbe-e8f9-4bab-9ae4-2279e79b7292" (UID: "e3281dbe-e8f9-4bab-9ae4-2279e79b7292"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:20.863817 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.863718 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dee11ef-9647-47ee-a4b1-686680256542-model-cache" pod "4dee11ef-9647-47ee-a4b1-686680256542" (UID: "4dee11ef-9647-47ee-a4b1-686680256542"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:20.863817 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.863756 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e044c02-0053-40e8-9f80-e806cd2d9ae5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e044c02-0053-40e8-9f80-e806cd2d9ae5-model-cache\") pod \"1e044c02-0053-40e8-9f80-e806cd2d9ae5\" (UID: \"1e044c02-0053-40e8-9f80-e806cd2d9ae5\") " Oct 30 22:04:20.863817 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.863790 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-tls-certs" pod "e3281dbe-e8f9-4bab-9ae4-2279e79b7292" (UID: "e3281dbe-e8f9-4bab-9ae4-2279e79b7292"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:20.864193 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.863855 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e044c02-0053-40e8-9f80-e806cd2d9ae5-kube-api-access-k5t4c" pod "1e044c02-0053-40e8-9f80-e806cd2d9ae5" (UID: "1e044c02-0053-40e8-9f80-e806cd2d9ae5"). InnerVolumeSpecName "kube-api-access-k5t4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:20.864193 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.864059 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e044c02-0053-40e8-9f80-e806cd2d9ae5-model-cache" pod "1e044c02-0053-40e8-9f80-e806cd2d9ae5" (UID: "1e044c02-0053-40e8-9f80-e806cd2d9ae5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:20.864321 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.864217 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e044c02-0053-40e8-9f80-e806cd2d9ae5-dshm" pod "1e044c02-0053-40e8-9f80-e806cd2d9ae5" (UID: "1e044c02-0053-40e8-9f80-e806cd2d9ae5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:20.864321 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.864224 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e044c02-0053-40e8-9f80-e806cd2d9ae5-model-cache\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:20.864321 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.864243 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e044c02-0053-40e8-9f80-e806cd2d9ae5-home\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:20.864321 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.864253 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-tls-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:20.864321 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.864263 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5t4c\" (UniqueName: \"kubernetes.io/projected/1e044c02-0053-40e8-9f80-e806cd2d9ae5-kube-api-access-k5t4c\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:20.864321 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.864272 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-model-cache\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:20.864321 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.864280 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dee11ef-9647-47ee-a4b1-686680256542-home\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:20.864321 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.864288 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-tmp-dir\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:20.864321 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.864288 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dee11ef-9647-47ee-a4b1-686680256542-dshm" pod "4dee11ef-9647-47ee-a4b1-686680256542" (UID: "4dee11ef-9647-47ee-a4b1-686680256542"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:20.864321 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.864296 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dee11ef-9647-47ee-a4b1-686680256542-tmp-dir\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:20.864930 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.864329 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e044c02-0053-40e8-9f80-e806cd2d9ae5-tmp-dir\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:20.864930 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.864373 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dee11ef-9647-47ee-a4b1-686680256542-model-cache\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:20.864930 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.864384 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-home\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:20.865094 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.864947 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-dshm" pod "e3281dbe-e8f9-4bab-9ae4-2279e79b7292" (UID: "e3281dbe-e8f9-4bab-9ae4-2279e79b7292"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:20.865881 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.865854 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dee11ef-9647-47ee-a4b1-686680256542-tls-certs" pod "4dee11ef-9647-47ee-a4b1-686680256542" (UID: "4dee11ef-9647-47ee-a4b1-686680256542"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:20.866207 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.866185 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e044c02-0053-40e8-9f80-e806cd2d9ae5-tls-certs" pod "1e044c02-0053-40e8-9f80-e806cd2d9ae5" (UID: "1e044c02-0053-40e8-9f80-e806cd2d9ae5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:20.866985 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.866954 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-kube-api-access-rfksb" pod "e3281dbe-e8f9-4bab-9ae4-2279e79b7292" (UID: "e3281dbe-e8f9-4bab-9ae4-2279e79b7292"). InnerVolumeSpecName "kube-api-access-rfksb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:20.867151 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.867129 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dee11ef-9647-47ee-a4b1-686680256542-kube-api-access-6qbtf" pod "4dee11ef-9647-47ee-a4b1-686680256542" (UID: "4dee11ef-9647-47ee-a4b1-686680256542"). InnerVolumeSpecName "kube-api-access-6qbtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:20.965468 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.965395 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e044c02-0053-40e8-9f80-e806cd2d9ae5-dshm\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:20.965468 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.965429 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dee11ef-9647-47ee-a4b1-686680256542-tls-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:20.965468 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.965444 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dee11ef-9647-47ee-a4b1-686680256542-dshm\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:20.965468 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.965459 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-dshm\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:20.965710 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.965473 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e044c02-0053-40e8-9f80-e806cd2d9ae5-tls-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:20.965710 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.965487 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfksb\" (UniqueName: \"kubernetes.io/projected/e3281dbe-e8f9-4bab-9ae4-2279e79b7292-kube-api-access-rfksb\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:20.965710 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:20.965503 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qbtf\" (UniqueName: \"kubernetes.io/projected/4dee11ef-9647-47ee-a4b1-686680256542-kube-api-access-6qbtf\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:21.014194 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:21.014163 4560 generic.go:356] "Generic (PLEG): container finished" podID="1e044c02-0053-40e8-9f80-e806cd2d9ae5" containerID="9dd25fa7195cbf6728318117ab0300f29ff576231a5b8c4958bf02c277efe16a" exitCode=0 Oct 30 22:04:21.014379 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:21.014218 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" Oct 30 22:04:21.014379 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:21.014245 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" event={"ID":"1e044c02-0053-40e8-9f80-e806cd2d9ae5","Type":"ContainerDied","Data":"9dd25fa7195cbf6728318117ab0300f29ff576231a5b8c4958bf02c277efe16a"} Oct 30 22:04:21.014379 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:21.014283 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg" event={"ID":"1e044c02-0053-40e8-9f80-e806cd2d9ae5","Type":"ContainerDied","Data":"4be83a0f00718aab11f3451ce360c2ad8431f09e63b504a0943361e0cf39f50b"} Oct 30 22:04:21.014379 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:21.014300 4560 scope.go:122] "RemoveContainer" containerID="9dd25fa7195cbf6728318117ab0300f29ff576231a5b8c4958bf02c277efe16a" Oct 30 22:04:21.015713 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:21.015688 4560 generic.go:356] "Generic (PLEG): container finished" podID="e3281dbe-e8f9-4bab-9ae4-2279e79b7292" containerID="44bb8b61144f12209f730c543869497701cc5a3c15dadb47a4a83a781912a01e" exitCode=0 Oct 30 22:04:21.015842 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:21.015726 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" Oct 30 22:04:21.015842 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:21.015763 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" event={"ID":"e3281dbe-e8f9-4bab-9ae4-2279e79b7292","Type":"ContainerDied","Data":"44bb8b61144f12209f730c543869497701cc5a3c15dadb47a4a83a781912a01e"} Oct 30 22:04:21.015842 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:21.015786 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc" event={"ID":"e3281dbe-e8f9-4bab-9ae4-2279e79b7292","Type":"ContainerDied","Data":"ec63e28ce9b2a8683a6fd9cf5581f37d6958d986f697abecacca629297dc354d"} Oct 30 22:04:21.017742 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:21.017709 4560 generic.go:356] "Generic (PLEG): container finished" podID="4dee11ef-9647-47ee-a4b1-686680256542" containerID="9b78d5b264887a2ae01f70c4ffe21587c242e470eead7b603c2191bdf76e4c55" exitCode=0 Oct 30 22:04:21.017854 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:21.017755 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" event={"ID":"4dee11ef-9647-47ee-a4b1-686680256542","Type":"ContainerDied","Data":"9b78d5b264887a2ae01f70c4ffe21587c242e470eead7b603c2191bdf76e4c55"} Oct 30 22:04:21.017854 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:21.017781 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" event={"ID":"4dee11ef-9647-47ee-a4b1-686680256542","Type":"ContainerDied","Data":"9d15a58f3e1b775a47fb459d58872ca4d74f9178804f8e6d60cf50418c4a1503"} Oct 30 22:04:21.017854 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:21.017844 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87" Oct 30 22:04:21.025956 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:21.025936 4560 scope.go:122] "RemoveContainer" containerID="9dd25fa7195cbf6728318117ab0300f29ff576231a5b8c4958bf02c277efe16a" Oct 30 22:04:21.026268 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:04:21.026245 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dd25fa7195cbf6728318117ab0300f29ff576231a5b8c4958bf02c277efe16a\": container with ID starting with 9dd25fa7195cbf6728318117ab0300f29ff576231a5b8c4958bf02c277efe16a not found: ID does not exist" containerID="9dd25fa7195cbf6728318117ab0300f29ff576231a5b8c4958bf02c277efe16a" Oct 30 22:04:21.026397 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:21.026277 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dd25fa7195cbf6728318117ab0300f29ff576231a5b8c4958bf02c277efe16a"} err="failed to get container status \"9dd25fa7195cbf6728318117ab0300f29ff576231a5b8c4958bf02c277efe16a\": rpc error: code = NotFound desc = could not find container \"9dd25fa7195cbf6728318117ab0300f29ff576231a5b8c4958bf02c277efe16a\": container with ID starting with 9dd25fa7195cbf6728318117ab0300f29ff576231a5b8c4958bf02c277efe16a not found: ID does not exist" Oct 30 22:04:21.026397 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:21.026298 4560 scope.go:122] "RemoveContainer" containerID="44bb8b61144f12209f730c543869497701cc5a3c15dadb47a4a83a781912a01e" Oct 30 22:04:21.034513 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:21.034495 4560 scope.go:122] "RemoveContainer" containerID="44bb8b61144f12209f730c543869497701cc5a3c15dadb47a4a83a781912a01e" Oct 30 22:04:21.034730 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:04:21.034713 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44bb8b61144f12209f730c543869497701cc5a3c15dadb47a4a83a781912a01e\": container with ID starting with 44bb8b61144f12209f730c543869497701cc5a3c15dadb47a4a83a781912a01e not found: ID does not exist" containerID="44bb8b61144f12209f730c543869497701cc5a3c15dadb47a4a83a781912a01e" Oct 30 22:04:21.034776 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:21.034739 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44bb8b61144f12209f730c543869497701cc5a3c15dadb47a4a83a781912a01e"} err="failed to get container status \"44bb8b61144f12209f730c543869497701cc5a3c15dadb47a4a83a781912a01e\": rpc error: code = NotFound desc = could not find container \"44bb8b61144f12209f730c543869497701cc5a3c15dadb47a4a83a781912a01e\": container with ID starting with 44bb8b61144f12209f730c543869497701cc5a3c15dadb47a4a83a781912a01e not found: ID does not exist" Oct 30 22:04:21.034776 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:21.034756 4560 scope.go:122] "RemoveContainer" containerID="9b78d5b264887a2ae01f70c4ffe21587c242e470eead7b603c2191bdf76e4c55" Oct 30 22:04:21.042436 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:21.042415 4560 scope.go:122] "RemoveContainer" containerID="9b78d5b264887a2ae01f70c4ffe21587c242e470eead7b603c2191bdf76e4c55" Oct 30 22:04:21.042682 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:04:21.042665 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b78d5b264887a2ae01f70c4ffe21587c242e470eead7b603c2191bdf76e4c55\": container with ID starting with 9b78d5b264887a2ae01f70c4ffe21587c242e470eead7b603c2191bdf76e4c55 not found: ID does not exist" containerID="9b78d5b264887a2ae01f70c4ffe21587c242e470eead7b603c2191bdf76e4c55" Oct 30 22:04:21.042735 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:21.042690 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b78d5b264887a2ae01f70c4ffe21587c242e470eead7b603c2191bdf76e4c55"} err="failed to get container status \"9b78d5b264887a2ae01f70c4ffe21587c242e470eead7b603c2191bdf76e4c55\": rpc error: code = NotFound desc = could not find container \"9b78d5b264887a2ae01f70c4ffe21587c242e470eead7b603c2191bdf76e4c55\": container with ID starting with 9b78d5b264887a2ae01f70c4ffe21587c242e470eead7b603c2191bdf76e4c55 not found: ID does not exist" Oct 30 22:04:21.160530 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:21.160491 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87"] Oct 30 22:04:21.165437 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:21.165410 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-ed76b8/simulator-w2-ed76b8-kserve-6f96c5c447-6qk87"] Oct 30 22:04:21.282467 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:21.282419 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc"] Oct 30 22:04:21.283514 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:21.283484 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-ed76b8/premium-w2-ed76b8-kserve-5586cc459d-4lnkc"] Oct 30 22:04:21.377881 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:21.377845 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg"] Oct 30 22:04:21.382376 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:21.382352 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-ed76b8/distinct-w2-ed76b8-kserve-7b6599cbd9-wtfrg"] Oct 30 22:04:22.574323 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:22.574291 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e044c02-0053-40e8-9f80-e806cd2d9ae5" path="/var/lib/kubelet/pods/1e044c02-0053-40e8-9f80-e806cd2d9ae5/volumes" Oct 30 22:04:22.574738 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:22.574713 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dee11ef-9647-47ee-a4b1-686680256542" path="/var/lib/kubelet/pods/4dee11ef-9647-47ee-a4b1-686680256542/volumes" Oct 30 22:04:22.575065 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:22.575053 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3281dbe-e8f9-4bab-9ae4-2279e79b7292" path="/var/lib/kubelet/pods/e3281dbe-e8f9-4bab-9ae4-2279e79b7292/volumes" Oct 30 22:04:28.060779 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:28.060742 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74fbce9f-qlm7g"] Oct 30 22:04:28.107444 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:28.107414 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0a1500a-5fqzs"] Oct 30 22:04:28.109262 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:28.109250 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0a1500a-5fqzs"] Oct 30 22:04:28.110812 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:28.110793 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74fbce9f-qlm7g"] Oct 30 22:04:28.574157 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:28.574123 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70bcada5-5c22-4367-9743-39a61a86a992" path="/var/lib/kubelet/pods/70bcada5-5c22-4367-9743-39a61a86a992/volumes" Oct 30 22:04:28.574558 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:28.574544 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f596e63b-dac7-425b-bec4-497f31cb8f26" path="/var/lib/kubelet/pods/f596e63b-dac7-425b-bec4-497f31cb8f26/volumes" Oct 30 22:04:31.297206 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.297175 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eba46b8-j2fq2"] Oct 30 22:04:31.297593 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.297533 4560 cpu_manager.go:414] "containerMap: removing container" podUID="3e61932e-01e1-4c1d-893e-632817e91104" containerName="revoke-keys" Oct 30 22:04:31.297593 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.297544 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e61932e-01e1-4c1d-893e-632817e91104" containerName="revoke-keys" Oct 30 22:04:31.297593 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.297564 4560 cpu_manager.go:414] "containerMap: removing container" podUID="4dee11ef-9647-47ee-a4b1-686680256542" containerName="main" Oct 30 22:04:31.297593 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.297572 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dee11ef-9647-47ee-a4b1-686680256542" containerName="main" Oct 30 22:04:31.297593 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.297583 4560 cpu_manager.go:414] "containerMap: removing container" podUID="405964a0-6ee3-4be8-8bac-0baaee68c93b" containerName="revoke-keys" Oct 30 22:04:31.297593 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.297588 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="405964a0-6ee3-4be8-8bac-0baaee68c93b" containerName="revoke-keys" Oct 30 22:04:31.297593 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.297597 4560 cpu_manager.go:414] "containerMap: removing container" podUID="e3281dbe-e8f9-4bab-9ae4-2279e79b7292" containerName="main" Oct 30 22:04:31.297815 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.297601 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3281dbe-e8f9-4bab-9ae4-2279e79b7292" containerName="main" Oct 30 22:04:31.297815 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.297609 4560 cpu_manager.go:414] "containerMap: removing container" podUID="7f4405ac-93f4-4a1d-a335-5990238c4294" containerName="istio-proxy" Oct 30 22:04:31.297815 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.297614 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f4405ac-93f4-4a1d-a335-5990238c4294" containerName="istio-proxy" Oct 30 22:04:31.297815 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.297623 4560 cpu_manager.go:414] "containerMap: removing container" podUID="1e044c02-0053-40e8-9f80-e806cd2d9ae5" containerName="main" Oct 30 22:04:31.297815 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.297627 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e044c02-0053-40e8-9f80-e806cd2d9ae5" containerName="main" Oct 30 22:04:31.297815 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.297644 4560 cpu_manager.go:414] "containerMap: removing container" podUID="7f4405ac-93f4-4a1d-a335-5990238c4294" containerName="istio-proxy" Oct 30 22:04:31.297815 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.297652 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f4405ac-93f4-4a1d-a335-5990238c4294" containerName="istio-proxy" Oct 30 22:04:31.297815 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.297706 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dee11ef-9647-47ee-a4b1-686680256542" containerName="main" Oct 30 22:04:31.297815 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.297713 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="405964a0-6ee3-4be8-8bac-0baaee68c93b" containerName="revoke-keys" Oct 30 22:04:31.297815 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.297721 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3281dbe-e8f9-4bab-9ae4-2279e79b7292" containerName="main" Oct 30 22:04:31.297815 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.297730 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f4405ac-93f4-4a1d-a335-5990238c4294" containerName="istio-proxy" Oct 30 22:04:31.297815 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.297738 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f4405ac-93f4-4a1d-a335-5990238c4294" containerName="istio-proxy" Oct 30 22:04:31.297815 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.297743 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e61932e-01e1-4c1d-893e-632817e91104" containerName="revoke-keys" Oct 30 22:04:31.297815 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.297750 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e044c02-0053-40e8-9f80-e806cd2d9ae5" containerName="main" Oct 30 22:04:31.302326 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.302308 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eba46b8-j2fq2" Oct 30 22:04:31.353667 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.353631 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7aca459f-5mh87"] Oct 30 22:04:31.357004 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.356985 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aca459f-5mh87" Oct 30 22:04:31.362205 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.362181 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/101812ce-bdc8-484c-ba45-a6c465341d0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4eba46b8-j2fq2\" (UID: \"101812ce-bdc8-484c-ba45-a6c465341d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eba46b8-j2fq2" Oct 30 22:04:31.362362 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.362318 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v66g\" (UniqueName: \"kubernetes.io/projected/101812ce-bdc8-484c-ba45-a6c465341d0d-kube-api-access-4v66g\") pod \"maas-api-revoke-sub-4eba46b8-j2fq2\" (UID: \"101812ce-bdc8-484c-ba45-a6c465341d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eba46b8-j2fq2" Oct 30 22:04:31.362430 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.362399 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/101812ce-bdc8-484c-ba45-a6c465341d0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4eba46b8-j2fq2\" (UID: \"101812ce-bdc8-484c-ba45-a6c465341d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eba46b8-j2fq2" Oct 30 22:04:31.398445 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.398417 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:04:31.400637 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.400612 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eba46b8-j2fq2"] Oct 30 22:04:31.401007 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.400985 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:04:31.401100 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.401045 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:04:31.401147 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.401135 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:04:31.455765 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.455735 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7aca459f-5mh87"] Oct 30 22:04:31.463010 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.462984 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v66g\" (UniqueName: \"kubernetes.io/projected/101812ce-bdc8-484c-ba45-a6c465341d0d-kube-api-access-4v66g\") pod \"maas-api-revoke-sub-4eba46b8-j2fq2\" (UID: \"101812ce-bdc8-484c-ba45-a6c465341d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eba46b8-j2fq2" Oct 30 22:04:31.463136 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.463017 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/101812ce-bdc8-484c-ba45-a6c465341d0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4eba46b8-j2fq2\" (UID: \"101812ce-bdc8-484c-ba45-a6c465341d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eba46b8-j2fq2" Oct 30 22:04:31.463136 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.463078 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82630713-ed8a-4ff9-b7f2-d85385bf69ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7aca459f-5mh87\" (UID: \"82630713-ed8a-4ff9-b7f2-d85385bf69ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aca459f-5mh87" Oct 30 22:04:31.463136 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.463123 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82630713-ed8a-4ff9-b7f2-d85385bf69ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7aca459f-5mh87\" (UID: \"82630713-ed8a-4ff9-b7f2-d85385bf69ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aca459f-5mh87" Oct 30 22:04:31.463273 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.463146 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrph6\" (UniqueName: \"kubernetes.io/projected/82630713-ed8a-4ff9-b7f2-d85385bf69ad-kube-api-access-rrph6\") pod \"maas-api-revoke-sub-7aca459f-5mh87\" (UID: \"82630713-ed8a-4ff9-b7f2-d85385bf69ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aca459f-5mh87" Oct 30 22:04:31.463273 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.463165 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/101812ce-bdc8-484c-ba45-a6c465341d0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4eba46b8-j2fq2\" (UID: \"101812ce-bdc8-484c-ba45-a6c465341d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eba46b8-j2fq2" Oct 30 22:04:31.463859 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.463837 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/101812ce-bdc8-484c-ba45-a6c465341d0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4eba46b8-j2fq2\" (UID: \"101812ce-bdc8-484c-ba45-a6c465341d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eba46b8-j2fq2" Oct 30 22:04:31.465605 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.465584 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/101812ce-bdc8-484c-ba45-a6c465341d0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4eba46b8-j2fq2\" (UID: \"101812ce-bdc8-484c-ba45-a6c465341d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eba46b8-j2fq2" Oct 30 22:04:31.521392 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.521366 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v66g\" (UniqueName: \"kubernetes.io/projected/101812ce-bdc8-484c-ba45-a6c465341d0d-kube-api-access-4v66g\") pod \"maas-api-revoke-sub-4eba46b8-j2fq2\" (UID: \"101812ce-bdc8-484c-ba45-a6c465341d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eba46b8-j2fq2" Oct 30 22:04:31.564186 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.564125 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82630713-ed8a-4ff9-b7f2-d85385bf69ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7aca459f-5mh87\" (UID: \"82630713-ed8a-4ff9-b7f2-d85385bf69ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aca459f-5mh87" Oct 30 22:04:31.564186 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.564179 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82630713-ed8a-4ff9-b7f2-d85385bf69ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7aca459f-5mh87\" (UID: \"82630713-ed8a-4ff9-b7f2-d85385bf69ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aca459f-5mh87" Oct 30 22:04:31.564331 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.564205 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrph6\" (UniqueName: \"kubernetes.io/projected/82630713-ed8a-4ff9-b7f2-d85385bf69ad-kube-api-access-rrph6\") pod \"maas-api-revoke-sub-7aca459f-5mh87\" (UID: \"82630713-ed8a-4ff9-b7f2-d85385bf69ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aca459f-5mh87" Oct 30 22:04:31.564839 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.564820 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82630713-ed8a-4ff9-b7f2-d85385bf69ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7aca459f-5mh87\" (UID: \"82630713-ed8a-4ff9-b7f2-d85385bf69ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aca459f-5mh87" Oct 30 22:04:31.566298 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.566273 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82630713-ed8a-4ff9-b7f2-d85385bf69ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7aca459f-5mh87\" (UID: \"82630713-ed8a-4ff9-b7f2-d85385bf69ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aca459f-5mh87" Oct 30 22:04:31.615113 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.615089 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eba46b8-j2fq2" Oct 30 22:04:31.622637 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.622612 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrph6\" (UniqueName: \"kubernetes.io/projected/82630713-ed8a-4ff9-b7f2-d85385bf69ad-kube-api-access-rrph6\") pod \"maas-api-revoke-sub-7aca459f-5mh87\" (UID: \"82630713-ed8a-4ff9-b7f2-d85385bf69ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aca459f-5mh87" Oct 30 22:04:31.672682 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.669557 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aca459f-5mh87" Oct 30 22:04:31.992181 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:31.992150 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eba46b8-j2fq2"] Oct 30 22:04:32.032327 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:04:32.032301 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod101812ce_bdc8_484c_ba45_a6c465341d0d.slice/crio-212326a3e0c8f713100abb6af1b05c432db6d0684233e9271285b45a5bc27f7c WatchSource:0}: Error finding container 212326a3e0c8f713100abb6af1b05c432db6d0684233e9271285b45a5bc27f7c: Status 404 returned error can't find the container with id 212326a3e0c8f713100abb6af1b05c432db6d0684233e9271285b45a5bc27f7c Oct 30 22:04:32.083007 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:32.082977 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eba46b8-j2fq2" event={"ID":"101812ce-bdc8-484c-ba45-a6c465341d0d","Type":"ContainerStarted","Data":"212326a3e0c8f713100abb6af1b05c432db6d0684233e9271285b45a5bc27f7c"} Oct 30 22:04:32.091907 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:32.091881 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7aca459f-5mh87"] Oct 30 22:04:32.097186 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:04:32.097163 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82630713_ed8a_4ff9_b7f2_d85385bf69ad.slice/crio-2538b04c3c5edb6491fcd49ec1f1afb39ab3da9dea57a8ee9c39c72ad6b30c98 WatchSource:0}: Error finding container 2538b04c3c5edb6491fcd49ec1f1afb39ab3da9dea57a8ee9c39c72ad6b30c98: Status 404 returned error can't find the container with id 2538b04c3c5edb6491fcd49ec1f1afb39ab3da9dea57a8ee9c39c72ad6b30c98 Oct 30 22:04:33.090129 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:33.090051 4560 generic.go:356] "Generic (PLEG): container finished" podID="101812ce-bdc8-484c-ba45-a6c465341d0d" containerID="a5b6c3fc2437bd23868a7706337e3f4d157ce37d7a95ea273d3e33efe9a80da2" exitCode=0 Oct 30 22:04:33.090129 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:33.090101 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eba46b8-j2fq2" event={"ID":"101812ce-bdc8-484c-ba45-a6c465341d0d","Type":"ContainerDied","Data":"a5b6c3fc2437bd23868a7706337e3f4d157ce37d7a95ea273d3e33efe9a80da2"} Oct 30 22:04:33.091743 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:33.091720 4560 generic.go:356] "Generic (PLEG): container finished" podID="82630713-ed8a-4ff9-b7f2-d85385bf69ad" containerID="7c063e470767e84d7ebc742ba246e9995c44197d99fb5ce3f4fc14b2ab73d8a0" exitCode=0 Oct 30 22:04:33.091838 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:33.091756 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aca459f-5mh87" event={"ID":"82630713-ed8a-4ff9-b7f2-d85385bf69ad","Type":"ContainerDied","Data":"7c063e470767e84d7ebc742ba246e9995c44197d99fb5ce3f4fc14b2ab73d8a0"} Oct 30 22:04:33.091838 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:33.091815 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aca459f-5mh87" event={"ID":"82630713-ed8a-4ff9-b7f2-d85385bf69ad","Type":"ContainerStarted","Data":"2538b04c3c5edb6491fcd49ec1f1afb39ab3da9dea57a8ee9c39c72ad6b30c98"} Oct 30 22:04:34.223468 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:34.223446 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eba46b8-j2fq2" Oct 30 22:04:34.258764 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:34.258741 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aca459f-5mh87" Oct 30 22:04:34.287223 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:34.287202 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/101812ce-bdc8-484c-ba45-a6c465341d0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/101812ce-bdc8-484c-ba45-a6c465341d0d-maas-api-xks-ca\") pod \"101812ce-bdc8-484c-ba45-a6c465341d0d\" (UID: \"101812ce-bdc8-484c-ba45-a6c465341d0d\") " Oct 30 22:04:34.287359 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:34.287233 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82630713-ed8a-4ff9-b7f2-d85385bf69ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82630713-ed8a-4ff9-b7f2-d85385bf69ad-maas-api-xks-ca\") pod \"82630713-ed8a-4ff9-b7f2-d85385bf69ad\" (UID: \"82630713-ed8a-4ff9-b7f2-d85385bf69ad\") " Oct 30 22:04:34.287359 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:34.287259 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82630713-ed8a-4ff9-b7f2-d85385bf69ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82630713-ed8a-4ff9-b7f2-d85385bf69ad-maas-api-openshift-ca\") pod \"82630713-ed8a-4ff9-b7f2-d85385bf69ad\" (UID: \"82630713-ed8a-4ff9-b7f2-d85385bf69ad\") " Oct 30 22:04:34.287359 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:34.287301 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82630713-ed8a-4ff9-b7f2-d85385bf69ad-kube-api-access-rrph6\" (UniqueName: \"kubernetes.io/projected/82630713-ed8a-4ff9-b7f2-d85385bf69ad-kube-api-access-rrph6\") pod \"82630713-ed8a-4ff9-b7f2-d85385bf69ad\" (UID: \"82630713-ed8a-4ff9-b7f2-d85385bf69ad\") " Oct 30 22:04:34.287359 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:34.287335 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/101812ce-bdc8-484c-ba45-a6c465341d0d-kube-api-access-4v66g\" (UniqueName: \"kubernetes.io/projected/101812ce-bdc8-484c-ba45-a6c465341d0d-kube-api-access-4v66g\") pod \"101812ce-bdc8-484c-ba45-a6c465341d0d\" (UID: \"101812ce-bdc8-484c-ba45-a6c465341d0d\") " Oct 30 22:04:34.287576 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:34.287501 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/101812ce-bdc8-484c-ba45-a6c465341d0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/101812ce-bdc8-484c-ba45-a6c465341d0d-maas-api-openshift-ca\") pod \"101812ce-bdc8-484c-ba45-a6c465341d0d\" (UID: \"101812ce-bdc8-484c-ba45-a6c465341d0d\") " Oct 30 22:04:34.287720 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:34.287685 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82630713-ed8a-4ff9-b7f2-d85385bf69ad-maas-api-openshift-ca" pod "82630713-ed8a-4ff9-b7f2-d85385bf69ad" (UID: "82630713-ed8a-4ff9-b7f2-d85385bf69ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:34.287950 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:34.287926 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/101812ce-bdc8-484c-ba45-a6c465341d0d-maas-api-openshift-ca" pod "101812ce-bdc8-484c-ba45-a6c465341d0d" (UID: "101812ce-bdc8-484c-ba45-a6c465341d0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:34.288048 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:34.287942 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82630713-ed8a-4ff9-b7f2-d85385bf69ad-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:34.289439 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:34.289414 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82630713-ed8a-4ff9-b7f2-d85385bf69ad-maas-api-xks-ca" pod "82630713-ed8a-4ff9-b7f2-d85385bf69ad" (UID: "82630713-ed8a-4ff9-b7f2-d85385bf69ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:34.289690 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:34.289672 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82630713-ed8a-4ff9-b7f2-d85385bf69ad-kube-api-access-rrph6" pod "82630713-ed8a-4ff9-b7f2-d85385bf69ad" (UID: "82630713-ed8a-4ff9-b7f2-d85385bf69ad"). InnerVolumeSpecName "kube-api-access-rrph6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:34.289834 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:34.289814 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/101812ce-bdc8-484c-ba45-a6c465341d0d-maas-api-xks-ca" pod "101812ce-bdc8-484c-ba45-a6c465341d0d" (UID: "101812ce-bdc8-484c-ba45-a6c465341d0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:34.289834 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:34.289826 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/101812ce-bdc8-484c-ba45-a6c465341d0d-kube-api-access-4v66g" pod "101812ce-bdc8-484c-ba45-a6c465341d0d" (UID: "101812ce-bdc8-484c-ba45-a6c465341d0d"). InnerVolumeSpecName "kube-api-access-4v66g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:34.389518 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:34.389446 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v66g\" (UniqueName: \"kubernetes.io/projected/101812ce-bdc8-484c-ba45-a6c465341d0d-kube-api-access-4v66g\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:34.389518 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:34.389474 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/101812ce-bdc8-484c-ba45-a6c465341d0d-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:34.389518 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:34.389488 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/101812ce-bdc8-484c-ba45-a6c465341d0d-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:34.389518 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:34.389497 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82630713-ed8a-4ff9-b7f2-d85385bf69ad-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:34.389518 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:34.389507 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrph6\" (UniqueName: \"kubernetes.io/projected/82630713-ed8a-4ff9-b7f2-d85385bf69ad-kube-api-access-rrph6\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:35.104563 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:35.104532 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aca459f-5mh87" Oct 30 22:04:35.104755 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:35.104529 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aca459f-5mh87" event={"ID":"82630713-ed8a-4ff9-b7f2-d85385bf69ad","Type":"ContainerDied","Data":"2538b04c3c5edb6491fcd49ec1f1afb39ab3da9dea57a8ee9c39c72ad6b30c98"} Oct 30 22:04:35.104755 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:35.104707 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2538b04c3c5edb6491fcd49ec1f1afb39ab3da9dea57a8ee9c39c72ad6b30c98" Oct 30 22:04:35.105964 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:35.105944 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eba46b8-j2fq2" Oct 30 22:04:35.105964 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:35.105955 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eba46b8-j2fq2" event={"ID":"101812ce-bdc8-484c-ba45-a6c465341d0d","Type":"ContainerDied","Data":"212326a3e0c8f713100abb6af1b05c432db6d0684233e9271285b45a5bc27f7c"} Oct 30 22:04:35.106091 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:35.105990 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="212326a3e0c8f713100abb6af1b05c432db6d0684233e9271285b45a5bc27f7c" Oct 30 22:04:38.693941 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:38.693911 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-87eb6a-gc5xh"] Oct 30 22:04:38.740406 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:38.740373 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-87eb6a-gc5xh"] Oct 30 22:04:40.574358 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:40.574318 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e61932e-01e1-4c1d-893e-632817e91104" path="/var/lib/kubelet/pods/3e61932e-01e1-4c1d-893e-632817e91104/volumes" Oct 30 22:04:47.544301 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:47.544260 4560 scope.go:122] "RemoveContainer" containerID="3b99dcb2363e26e290ff2eb60c251d7e9893f5a9d3719dbe840c2c44bbe716fb" Oct 30 22:04:47.553796 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:47.553768 4560 scope.go:122] "RemoveContainer" containerID="711e8abb5e1d8ae1e5138ab28e1490859b8e300fea805e05659185181c557c7a" Oct 30 22:04:47.565907 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:47.565881 4560 scope.go:122] "RemoveContainer" containerID="c03d1e618d74c9b8ba2bcbf41721d61bab6e16df6402e21677ba65d80d2c025b" Oct 30 22:04:47.574559 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:47.574535 4560 scope.go:122] "RemoveContainer" containerID="a0c831c6dd934318d9501c1a9fb0a39313872fa2b124d667662c06a09828b624" Oct 30 22:04:47.583519 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:47.583494 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-afbe5b-qpvcj"] Oct 30 22:04:47.583873 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:47.583855 4560 cpu_manager.go:414] "containerMap: removing container" podUID="82630713-ed8a-4ff9-b7f2-d85385bf69ad" containerName="revoke-keys" Oct 30 22:04:47.583925 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:47.583874 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82630713-ed8a-4ff9-b7f2-d85385bf69ad" containerName="revoke-keys" Oct 30 22:04:47.583925 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:47.583892 4560 cpu_manager.go:414] "containerMap: removing container" podUID="101812ce-bdc8-484c-ba45-a6c465341d0d" containerName="revoke-keys" Oct 30 22:04:47.583925 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:47.583898 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="101812ce-bdc8-484c-ba45-a6c465341d0d" containerName="revoke-keys" Oct 30 22:04:47.584023 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:47.583980 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="82630713-ed8a-4ff9-b7f2-d85385bf69ad" containerName="revoke-keys" Oct 30 22:04:47.584023 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:47.583990 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="101812ce-bdc8-484c-ba45-a6c465341d0d" containerName="revoke-keys" Oct 30 22:04:47.588565 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:47.588542 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-afbe5b-qpvcj" Oct 30 22:04:47.651685 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:47.651652 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:04:47.652239 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:47.652215 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:04:47.652391 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:47.652262 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:04:47.654824 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:47.654793 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-afbe5b-qpvcj"] Oct 30 22:04:47.682870 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:47.682840 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:04:47.726230 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:47.726194 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23ddc9db-728d-4e3c-a147-0eb3e070ad43-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-afbe5b-qpvcj\" (UID: \"23ddc9db-728d-4e3c-a147-0eb3e070ad43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-afbe5b-qpvcj" Oct 30 22:04:47.726506 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:47.726274 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h78sl\" (UniqueName: \"kubernetes.io/projected/23ddc9db-728d-4e3c-a147-0eb3e070ad43-kube-api-access-h78sl\") pod \"maas-api-revoke-keys-e2e-worker-w0-afbe5b-qpvcj\" (UID: \"23ddc9db-728d-4e3c-a147-0eb3e070ad43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-afbe5b-qpvcj" Oct 30 22:04:47.726629 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:47.726506 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23ddc9db-728d-4e3c-a147-0eb3e070ad43-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-afbe5b-qpvcj\" (UID: \"23ddc9db-728d-4e3c-a147-0eb3e070ad43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-afbe5b-qpvcj" Oct 30 22:04:47.827816 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:47.827727 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23ddc9db-728d-4e3c-a147-0eb3e070ad43-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-afbe5b-qpvcj\" (UID: \"23ddc9db-728d-4e3c-a147-0eb3e070ad43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-afbe5b-qpvcj" Oct 30 22:04:47.827816 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:47.827800 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23ddc9db-728d-4e3c-a147-0eb3e070ad43-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-afbe5b-qpvcj\" (UID: \"23ddc9db-728d-4e3c-a147-0eb3e070ad43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-afbe5b-qpvcj" Oct 30 22:04:47.828021 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:47.827866 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h78sl\" (UniqueName: \"kubernetes.io/projected/23ddc9db-728d-4e3c-a147-0eb3e070ad43-kube-api-access-h78sl\") pod \"maas-api-revoke-keys-e2e-worker-w0-afbe5b-qpvcj\" (UID: \"23ddc9db-728d-4e3c-a147-0eb3e070ad43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-afbe5b-qpvcj" Oct 30 22:04:47.828594 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:47.828557 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23ddc9db-728d-4e3c-a147-0eb3e070ad43-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-afbe5b-qpvcj\" (UID: \"23ddc9db-728d-4e3c-a147-0eb3e070ad43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-afbe5b-qpvcj" Oct 30 22:04:47.830146 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:47.830127 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23ddc9db-728d-4e3c-a147-0eb3e070ad43-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-afbe5b-qpvcj\" (UID: \"23ddc9db-728d-4e3c-a147-0eb3e070ad43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-afbe5b-qpvcj" Oct 30 22:04:47.886553 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:47.886511 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h78sl\" (UniqueName: \"kubernetes.io/projected/23ddc9db-728d-4e3c-a147-0eb3e070ad43-kube-api-access-h78sl\") pod \"maas-api-revoke-keys-e2e-worker-w0-afbe5b-qpvcj\" (UID: \"23ddc9db-728d-4e3c-a147-0eb3e070ad43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-afbe5b-qpvcj" Oct 30 22:04:47.903231 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:47.903198 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-afbe5b-qpvcj" Oct 30 22:04:48.278618 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:48.278570 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-afbe5b-qpvcj"] Oct 30 22:04:48.284310 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:04:48.284274 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23ddc9db_728d_4e3c_a147_0eb3e070ad43.slice/crio-4a10f6c8e20de7191dfd46fc6d9bbcdcfaee8e4210c81ff949181f927611ddf8 WatchSource:0}: Error finding container 4a10f6c8e20de7191dfd46fc6d9bbcdcfaee8e4210c81ff949181f927611ddf8: Status 404 returned error can't find the container with id 4a10f6c8e20de7191dfd46fc6d9bbcdcfaee8e4210c81ff949181f927611ddf8 Oct 30 22:04:49.186566 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:49.186532 4560 generic.go:356] "Generic (PLEG): container finished" podID="23ddc9db-728d-4e3c-a147-0eb3e070ad43" containerID="00b6a84016c9bdcd1aa3dc1ca2cf2f65f9a5aef36a25e7b28b67eb3eb9009163" exitCode=0 Oct 30 22:04:49.186984 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:49.186568 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-afbe5b-qpvcj" event={"ID":"23ddc9db-728d-4e3c-a147-0eb3e070ad43","Type":"ContainerDied","Data":"00b6a84016c9bdcd1aa3dc1ca2cf2f65f9a5aef36a25e7b28b67eb3eb9009163"} Oct 30 22:04:49.186984 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:49.186617 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-afbe5b-qpvcj" event={"ID":"23ddc9db-728d-4e3c-a147-0eb3e070ad43","Type":"ContainerStarted","Data":"4a10f6c8e20de7191dfd46fc6d9bbcdcfaee8e4210c81ff949181f927611ddf8"} Oct 30 22:04:49.685105 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:04:49.685068 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:04:50.324514 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:50.324489 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-afbe5b-qpvcj" Oct 30 22:04:50.461422 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:50.461329 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23ddc9db-728d-4e3c-a147-0eb3e070ad43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23ddc9db-728d-4e3c-a147-0eb3e070ad43-maas-api-xks-ca\") pod \"23ddc9db-728d-4e3c-a147-0eb3e070ad43\" (UID: \"23ddc9db-728d-4e3c-a147-0eb3e070ad43\") " Oct 30 22:04:50.461597 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:50.461485 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23ddc9db-728d-4e3c-a147-0eb3e070ad43-kube-api-access-h78sl\" (UniqueName: \"kubernetes.io/projected/23ddc9db-728d-4e3c-a147-0eb3e070ad43-kube-api-access-h78sl\") pod \"23ddc9db-728d-4e3c-a147-0eb3e070ad43\" (UID: \"23ddc9db-728d-4e3c-a147-0eb3e070ad43\") " Oct 30 22:04:50.461597 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:50.461515 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23ddc9db-728d-4e3c-a147-0eb3e070ad43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23ddc9db-728d-4e3c-a147-0eb3e070ad43-maas-api-openshift-ca\") pod \"23ddc9db-728d-4e3c-a147-0eb3e070ad43\" (UID: \"23ddc9db-728d-4e3c-a147-0eb3e070ad43\") " Oct 30 22:04:50.461994 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:50.461954 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23ddc9db-728d-4e3c-a147-0eb3e070ad43-maas-api-openshift-ca" pod "23ddc9db-728d-4e3c-a147-0eb3e070ad43" (UID: "23ddc9db-728d-4e3c-a147-0eb3e070ad43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:50.463793 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:50.463764 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23ddc9db-728d-4e3c-a147-0eb3e070ad43-kube-api-access-h78sl" pod "23ddc9db-728d-4e3c-a147-0eb3e070ad43" (UID: "23ddc9db-728d-4e3c-a147-0eb3e070ad43"). InnerVolumeSpecName "kube-api-access-h78sl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:50.463947 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:50.463786 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23ddc9db-728d-4e3c-a147-0eb3e070ad43-maas-api-xks-ca" pod "23ddc9db-728d-4e3c-a147-0eb3e070ad43" (UID: "23ddc9db-728d-4e3c-a147-0eb3e070ad43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:50.566846 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:50.565017 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23ddc9db-728d-4e3c-a147-0eb3e070ad43-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:50.566846 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:50.565063 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h78sl\" (UniqueName: \"kubernetes.io/projected/23ddc9db-728d-4e3c-a147-0eb3e070ad43-kube-api-access-h78sl\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:50.566846 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:50.565083 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23ddc9db-728d-4e3c-a147-0eb3e070ad43-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:50.729483 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:50.729451 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ed76b8-7fnl9"] Oct 30 22:04:50.738928 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:50.738904 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ed76b8-7fnl9"] Oct 30 22:04:51.200312 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:51.200228 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-afbe5b-qpvcj" event={"ID":"23ddc9db-728d-4e3c-a147-0eb3e070ad43","Type":"ContainerDied","Data":"4a10f6c8e20de7191dfd46fc6d9bbcdcfaee8e4210c81ff949181f927611ddf8"} Oct 30 22:04:51.200312 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:51.200242 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-afbe5b-qpvcj" Oct 30 22:04:51.200312 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:51.200284 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a10f6c8e20de7191dfd46fc6d9bbcdcfaee8e4210c81ff949181f927611ddf8" Oct 30 22:04:51.937413 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:51.937376 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w"] Oct 30 22:04:51.942955 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:51.942929 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd"] Oct 30 22:04:51.983608 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:51.983583 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9"] Oct 30 22:04:51.986682 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:51.986651 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx"] Oct 30 22:04:52.574893 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:52.574859 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="405964a0-6ee3-4be8-8bac-0baaee68c93b" path="/var/lib/kubelet/pods/405964a0-6ee3-4be8-8bac-0baaee68c93b/volumes" Oct 30 22:04:55.969544 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:55.969504 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:04:59.241690 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:59.241659 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-afbe5b-85d55db988-8c4wd"] Oct 30 22:04:59.340194 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:59.340161 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-afbe5b-646894cd95-h5tqj"] Oct 30 22:04:59.563568 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:59.563476 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh"] Oct 30 22:04:59.563784 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:04:59.563757 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" podUID="b007cff0-0922-4fbf-b4fc-10aaf629a952" containerName="istio-proxy" containerID="cri-o://84fb0557cf8858144316819eed9efcf1a3b0c8f296164895a19cba8e2f427acc" gracePeriod=30 Oct 30 22:05:01.316400 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:01.316350 4560 patch_prober.go:29] interesting pod/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:05:01.316807 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:01.316418 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" podUID="b007cff0-0922-4fbf-b4fc-10aaf629a952" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:05:04.242469 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:04.242408 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-afbe5b-85d55db988-8c4wd" podUID="024e6f83-289f-499a-be2d-5aed8a5a6c6d" containerName="payload-processing" containerID="cri-o://2660ad319e7c9efa9255615da5d042b7aa05f14a25ba1091e2ae4464ba32134a" gracePeriod=25 Oct 30 22:05:04.341506 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:04.341415 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-afbe5b-646894cd95-h5tqj" podUID="1807c151-3102-4158-b028-d7dd697f4ad6" containerName="payload-pre-processing" containerID="cri-o://6571f2bf6dceb7f69dac253f0b49495f4b43dd0f9035ebec675e8bed5206716f" gracePeriod=25 Oct 30 22:05:04.819199 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:04.819168 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" Oct 30 22:05:04.928010 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:04.927926 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b007cff0-0922-4fbf-b4fc-10aaf629a952-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b007cff0-0922-4fbf-b4fc-10aaf629a952-istiod-ca-cert\") pod \"b007cff0-0922-4fbf-b4fc-10aaf629a952\" (UID: \"b007cff0-0922-4fbf-b4fc-10aaf629a952\") " Oct 30 22:05:04.928010 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:04.927963 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b007cff0-0922-4fbf-b4fc-10aaf629a952-istio-token\" (UniqueName: \"kubernetes.io/projected/b007cff0-0922-4fbf-b4fc-10aaf629a952-istio-token\") pod \"b007cff0-0922-4fbf-b4fc-10aaf629a952\" (UID: \"b007cff0-0922-4fbf-b4fc-10aaf629a952\") " Oct 30 22:05:04.928236 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:04.928057 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b007cff0-0922-4fbf-b4fc-10aaf629a952-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b007cff0-0922-4fbf-b4fc-10aaf629a952-credential-socket\") pod \"b007cff0-0922-4fbf-b4fc-10aaf629a952\" (UID: \"b007cff0-0922-4fbf-b4fc-10aaf629a952\") " Oct 30 22:05:04.928236 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:04.928103 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b007cff0-0922-4fbf-b4fc-10aaf629a952-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b007cff0-0922-4fbf-b4fc-10aaf629a952-istio-envoy\") pod \"b007cff0-0922-4fbf-b4fc-10aaf629a952\" (UID: \"b007cff0-0922-4fbf-b4fc-10aaf629a952\") " Oct 30 22:05:04.928236 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:04.928159 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b007cff0-0922-4fbf-b4fc-10aaf629a952-kube-api-access-t7pvq\" (UniqueName: \"kubernetes.io/projected/b007cff0-0922-4fbf-b4fc-10aaf629a952-kube-api-access-t7pvq\") pod \"b007cff0-0922-4fbf-b4fc-10aaf629a952\" (UID: \"b007cff0-0922-4fbf-b4fc-10aaf629a952\") " Oct 30 22:05:04.928236 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:04.928195 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b007cff0-0922-4fbf-b4fc-10aaf629a952-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b007cff0-0922-4fbf-b4fc-10aaf629a952-istio-podinfo\") pod \"b007cff0-0922-4fbf-b4fc-10aaf629a952\" (UID: \"b007cff0-0922-4fbf-b4fc-10aaf629a952\") " Oct 30 22:05:04.928500 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:04.928242 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b007cff0-0922-4fbf-b4fc-10aaf629a952-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b007cff0-0922-4fbf-b4fc-10aaf629a952-istio-data\") pod \"b007cff0-0922-4fbf-b4fc-10aaf629a952\" (UID: \"b007cff0-0922-4fbf-b4fc-10aaf629a952\") " Oct 30 22:05:04.928500 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:04.928281 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b007cff0-0922-4fbf-b4fc-10aaf629a952-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b007cff0-0922-4fbf-b4fc-10aaf629a952-workload-socket\") pod \"b007cff0-0922-4fbf-b4fc-10aaf629a952\" (UID: \"b007cff0-0922-4fbf-b4fc-10aaf629a952\") " Oct 30 22:05:04.928500 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:04.928324 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b007cff0-0922-4fbf-b4fc-10aaf629a952-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b007cff0-0922-4fbf-b4fc-10aaf629a952-workload-certs\") pod \"b007cff0-0922-4fbf-b4fc-10aaf629a952\" (UID: \"b007cff0-0922-4fbf-b4fc-10aaf629a952\") " Oct 30 22:05:04.928500 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:04.928407 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b007cff0-0922-4fbf-b4fc-10aaf629a952-istiod-ca-cert" pod "b007cff0-0922-4fbf-b4fc-10aaf629a952" (UID: "b007cff0-0922-4fbf-b4fc-10aaf629a952"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:04.928500 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:04.928403 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b007cff0-0922-4fbf-b4fc-10aaf629a952-credential-socket" pod "b007cff0-0922-4fbf-b4fc-10aaf629a952" (UID: "b007cff0-0922-4fbf-b4fc-10aaf629a952"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:04.928778 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:04.928610 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b007cff0-0922-4fbf-b4fc-10aaf629a952-istio-data" pod "b007cff0-0922-4fbf-b4fc-10aaf629a952" (UID: "b007cff0-0922-4fbf-b4fc-10aaf629a952"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:04.928778 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:04.928672 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b007cff0-0922-4fbf-b4fc-10aaf629a952-credential-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:04.928778 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:04.928693 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b007cff0-0922-4fbf-b4fc-10aaf629a952-istiod-ca-cert\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:04.928778 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:04.928675 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b007cff0-0922-4fbf-b4fc-10aaf629a952-workload-socket" pod "b007cff0-0922-4fbf-b4fc-10aaf629a952" (UID: "b007cff0-0922-4fbf-b4fc-10aaf629a952"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:04.928994 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:04.928814 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b007cff0-0922-4fbf-b4fc-10aaf629a952-workload-certs" pod "b007cff0-0922-4fbf-b4fc-10aaf629a952" (UID: "b007cff0-0922-4fbf-b4fc-10aaf629a952"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:04.930371 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:04.930333 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b007cff0-0922-4fbf-b4fc-10aaf629a952-istio-envoy" pod "b007cff0-0922-4fbf-b4fc-10aaf629a952" (UID: "b007cff0-0922-4fbf-b4fc-10aaf629a952"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:04.930501 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:04.930453 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b007cff0-0922-4fbf-b4fc-10aaf629a952-istio-podinfo" pod "b007cff0-0922-4fbf-b4fc-10aaf629a952" (UID: "b007cff0-0922-4fbf-b4fc-10aaf629a952"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:05:04.930720 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:04.930703 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b007cff0-0922-4fbf-b4fc-10aaf629a952-kube-api-access-t7pvq" pod "b007cff0-0922-4fbf-b4fc-10aaf629a952" (UID: "b007cff0-0922-4fbf-b4fc-10aaf629a952"). InnerVolumeSpecName "kube-api-access-t7pvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:04.930766 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:04.930754 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b007cff0-0922-4fbf-b4fc-10aaf629a952-istio-token" pod "b007cff0-0922-4fbf-b4fc-10aaf629a952" (UID: "b007cff0-0922-4fbf-b4fc-10aaf629a952"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:05.030079 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:05.030049 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b007cff0-0922-4fbf-b4fc-10aaf629a952-istio-envoy\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:05.030079 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:05.030079 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7pvq\" (UniqueName: \"kubernetes.io/projected/b007cff0-0922-4fbf-b4fc-10aaf629a952-kube-api-access-t7pvq\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:05.030079 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:05.030096 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b007cff0-0922-4fbf-b4fc-10aaf629a952-istio-podinfo\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:05.030373 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:05.030109 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b007cff0-0922-4fbf-b4fc-10aaf629a952-istio-data\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:05.030373 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:05.030121 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b007cff0-0922-4fbf-b4fc-10aaf629a952-workload-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:05.030373 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:05.030133 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b007cff0-0922-4fbf-b4fc-10aaf629a952-workload-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:05.030373 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:05.030144 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b007cff0-0922-4fbf-b4fc-10aaf629a952-istio-token\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:05.292542 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:05.292503 4560 generic.go:356] "Generic (PLEG): container finished" podID="b007cff0-0922-4fbf-b4fc-10aaf629a952" containerID="84fb0557cf8858144316819eed9efcf1a3b0c8f296164895a19cba8e2f427acc" exitCode=0 Oct 30 22:05:05.293019 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:05.292557 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" Oct 30 22:05:05.293019 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:05.292558 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" event={"ID":"b007cff0-0922-4fbf-b4fc-10aaf629a952","Type":"ContainerDied","Data":"84fb0557cf8858144316819eed9efcf1a3b0c8f296164895a19cba8e2f427acc"} Oct 30 22:05:05.293019 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:05.292625 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh" event={"ID":"b007cff0-0922-4fbf-b4fc-10aaf629a952","Type":"ContainerDied","Data":"6a6808889dfa988a43ba69c15db6064311e77efbe69134d6105667c1ff933400"} Oct 30 22:05:05.293019 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:05.292655 4560 scope.go:122] "RemoveContainer" containerID="84fb0557cf8858144316819eed9efcf1a3b0c8f296164895a19cba8e2f427acc" Oct 30 22:05:05.303042 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:05.303025 4560 scope.go:122] "RemoveContainer" containerID="84fb0557cf8858144316819eed9efcf1a3b0c8f296164895a19cba8e2f427acc" Oct 30 22:05:05.303358 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:05:05.303317 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84fb0557cf8858144316819eed9efcf1a3b0c8f296164895a19cba8e2f427acc\": container with ID starting with 84fb0557cf8858144316819eed9efcf1a3b0c8f296164895a19cba8e2f427acc not found: ID does not exist" containerID="84fb0557cf8858144316819eed9efcf1a3b0c8f296164895a19cba8e2f427acc" Oct 30 22:05:05.303407 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:05.303377 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84fb0557cf8858144316819eed9efcf1a3b0c8f296164895a19cba8e2f427acc"} err="failed to get container status \"84fb0557cf8858144316819eed9efcf1a3b0c8f296164895a19cba8e2f427acc\": rpc error: code = NotFound desc = could not find container \"84fb0557cf8858144316819eed9efcf1a3b0c8f296164895a19cba8e2f427acc\": container with ID starting with 84fb0557cf8858144316819eed9efcf1a3b0c8f296164895a19cba8e2f427acc not found: ID does not exist" Oct 30 22:05:05.317296 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:05.317265 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35b8a54f-zq6zv"] Oct 30 22:05:05.317866 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:05.317796 4560 cpu_manager.go:414] "containerMap: removing container" podUID="23ddc9db-728d-4e3c-a147-0eb3e070ad43" containerName="revoke-keys" Oct 30 22:05:05.317866 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:05.317820 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23ddc9db-728d-4e3c-a147-0eb3e070ad43" containerName="revoke-keys" Oct 30 22:05:05.317866 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:05.317865 4560 cpu_manager.go:414] "containerMap: removing container" podUID="b007cff0-0922-4fbf-b4fc-10aaf629a952" containerName="istio-proxy" Oct 30 22:05:05.318071 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:05.317871 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b007cff0-0922-4fbf-b4fc-10aaf629a952" containerName="istio-proxy" Oct 30 22:05:05.318071 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:05.317941 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="b007cff0-0922-4fbf-b4fc-10aaf629a952" containerName="istio-proxy" Oct 30 22:05:05.318071 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:05.317952 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="23ddc9db-728d-4e3c-a147-0eb3e070ad43" containerName="revoke-keys" Oct 30 22:05:05.322458 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:05.322441 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b8a54f-zq6zv" Oct 30 22:05:05.373394 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:05.373369 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:05:05.376538 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:05.376387 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:05:05.376720 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:05.376660 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:05:05.377464 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:05.377441 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:05:05.379599 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:05.379568 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35b8a54f-zq6zv"] Oct 30 22:05:05.434949 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:05.434919 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef6aad01-3c06-4cb5-9e33-a43d39a5b917-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35b8a54f-zq6zv\" (UID: \"ef6aad01-3c06-4cb5-9e33-a43d39a5b917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b8a54f-zq6zv" Oct 30 22:05:05.435106 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:05.435029 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n7pw\" (UniqueName: \"kubernetes.io/projected/ef6aad01-3c06-4cb5-9e33-a43d39a5b917-kube-api-access-7n7pw\") pod \"maas-api-revoke-sub-35b8a54f-zq6zv\" (UID: \"ef6aad01-3c06-4cb5-9e33-a43d39a5b917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b8a54f-zq6zv" Oct 30 22:05:05.435106 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:05.435097 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef6aad01-3c06-4cb5-9e33-a43d39a5b917-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35b8a54f-zq6zv\" (UID: \"ef6aad01-3c06-4cb5-9e33-a43d39a5b917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b8a54f-zq6zv" Oct 30 22:05:05.455215 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:05.455186 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh"] Oct 30 22:05:05.458437 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:05.458414 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ed76b8-openshift-default-68765db7c-g86nh"] Oct 30 22:05:05.536002 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:05.535967 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n7pw\" (UniqueName: \"kubernetes.io/projected/ef6aad01-3c06-4cb5-9e33-a43d39a5b917-kube-api-access-7n7pw\") pod \"maas-api-revoke-sub-35b8a54f-zq6zv\" (UID: \"ef6aad01-3c06-4cb5-9e33-a43d39a5b917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b8a54f-zq6zv" Oct 30 22:05:05.536189 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:05.536033 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef6aad01-3c06-4cb5-9e33-a43d39a5b917-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35b8a54f-zq6zv\" (UID: \"ef6aad01-3c06-4cb5-9e33-a43d39a5b917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b8a54f-zq6zv" Oct 30 22:05:05.536189 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:05.536059 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef6aad01-3c06-4cb5-9e33-a43d39a5b917-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35b8a54f-zq6zv\" (UID: \"ef6aad01-3c06-4cb5-9e33-a43d39a5b917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b8a54f-zq6zv" Oct 30 22:05:05.536762 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:05.536743 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef6aad01-3c06-4cb5-9e33-a43d39a5b917-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35b8a54f-zq6zv\" (UID: \"ef6aad01-3c06-4cb5-9e33-a43d39a5b917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b8a54f-zq6zv" Oct 30 22:05:05.538274 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:05.538258 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef6aad01-3c06-4cb5-9e33-a43d39a5b917-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35b8a54f-zq6zv\" (UID: \"ef6aad01-3c06-4cb5-9e33-a43d39a5b917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b8a54f-zq6zv" Oct 30 22:05:05.594604 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:05.594529 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n7pw\" (UniqueName: \"kubernetes.io/projected/ef6aad01-3c06-4cb5-9e33-a43d39a5b917-kube-api-access-7n7pw\") pod \"maas-api-revoke-sub-35b8a54f-zq6zv\" (UID: \"ef6aad01-3c06-4cb5-9e33-a43d39a5b917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b8a54f-zq6zv" Oct 30 22:05:05.634136 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:05.634105 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b8a54f-zq6zv" Oct 30 22:05:06.012450 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:06.012411 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35b8a54f-zq6zv"] Oct 30 22:05:06.018189 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:05:06.018157 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef6aad01_3c06_4cb5_9e33_a43d39a5b917.slice/crio-720885257a9497eadf255a8a6a8ce6942491434b910a64d934365adf0b893cac WatchSource:0}: Error finding container 720885257a9497eadf255a8a6a8ce6942491434b910a64d934365adf0b893cac: Status 404 returned error can't find the container with id 720885257a9497eadf255a8a6a8ce6942491434b910a64d934365adf0b893cac Oct 30 22:05:06.300939 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:06.300853 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b8a54f-zq6zv" event={"ID":"ef6aad01-3c06-4cb5-9e33-a43d39a5b917","Type":"ContainerStarted","Data":"720885257a9497eadf255a8a6a8ce6942491434b910a64d934365adf0b893cac"} Oct 30 22:05:06.574597 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:06.574498 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b007cff0-0922-4fbf-b4fc-10aaf629a952" path="/var/lib/kubelet/pods/b007cff0-0922-4fbf-b4fc-10aaf629a952/volumes" Oct 30 22:05:06.957097 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:06.957052 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" podUID="9e2d7535-075d-4211-b814-35d6abff7605" containerName="main" containerID="cri-o://464a950cb3eb7661e8c2a30c1b06fd05892a0b474bffc35fdfd16a2bf907abb6" gracePeriod=45 Oct 30 22:05:06.961494 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:06.961458 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" podUID="971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d" containerName="main" containerID="cri-o://3b09ed402efdf45ad09ae6b962dcff9a0c1711d4707e88e95372afeac16abebc" gracePeriod=45 Oct 30 22:05:07.002198 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.002161 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" podUID="a59a0fc3-8316-47eb-a42e-552d202b1ab7" containerName="main" containerID="cri-o://3d9ac6e9d0d2264fe8c1ecfa3b11164581832dce26b6b645f05d81fb438adca5" gracePeriod=45 Oct 30 22:05:07.005107 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.005074 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" podUID="90bfb1e4-9526-40f6-aed4-ee46c2d86471" containerName="main" containerID="cri-o://83471276efe32bda9d7935fc7b28bf80bc1c1a901efd72e572720261129c37bb" gracePeriod=45 Oct 30 22:05:07.131592 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.131548 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" podUID="a59a0fc3-8316-47eb-a42e-552d202b1ab7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": dial tcp 10.133.0.42:8000: connect: connection refused" Oct 30 22:05:07.258651 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.258611 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" Oct 30 22:05:07.270077 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.270044 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" Oct 30 22:05:07.310799 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.310678 4560 generic.go:356] "Generic (PLEG): container finished" podID="971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d" containerID="3b09ed402efdf45ad09ae6b962dcff9a0c1711d4707e88e95372afeac16abebc" exitCode=0 Oct 30 22:05:07.310799 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.310722 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" Oct 30 22:05:07.310799 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.310765 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" event={"ID":"971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d","Type":"ContainerDied","Data":"3b09ed402efdf45ad09ae6b962dcff9a0c1711d4707e88e95372afeac16abebc"} Oct 30 22:05:07.311256 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.310814 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd" event={"ID":"971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d","Type":"ContainerDied","Data":"2fc84004c095b341be3c9751697b2a36f3988d4a44abd09981bcb1e7157ed978"} Oct 30 22:05:07.311256 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.310844 4560 scope.go:122] "RemoveContainer" containerID="3b09ed402efdf45ad09ae6b962dcff9a0c1711d4707e88e95372afeac16abebc" Oct 30 22:05:07.312323 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.312300 4560 generic.go:356] "Generic (PLEG): container finished" podID="90bfb1e4-9526-40f6-aed4-ee46c2d86471" containerID="83471276efe32bda9d7935fc7b28bf80bc1c1a901efd72e572720261129c37bb" exitCode=0 Oct 30 22:05:07.312453 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.312379 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" event={"ID":"90bfb1e4-9526-40f6-aed4-ee46c2d86471","Type":"ContainerDied","Data":"83471276efe32bda9d7935fc7b28bf80bc1c1a901efd72e572720261129c37bb"} Oct 30 22:05:07.312587 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.312571 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" Oct 30 22:05:07.314047 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.314024 4560 generic.go:356] "Generic (PLEG): container finished" podID="a59a0fc3-8316-47eb-a42e-552d202b1ab7" containerID="3d9ac6e9d0d2264fe8c1ecfa3b11164581832dce26b6b645f05d81fb438adca5" exitCode=0 Oct 30 22:05:07.314141 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.314094 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" event={"ID":"a59a0fc3-8316-47eb-a42e-552d202b1ab7","Type":"ContainerDied","Data":"3d9ac6e9d0d2264fe8c1ecfa3b11164581832dce26b6b645f05d81fb438adca5"} Oct 30 22:05:07.315970 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.315896 4560 generic.go:356] "Generic (PLEG): container finished" podID="ef6aad01-3c06-4cb5-9e33-a43d39a5b917" containerID="9ddaeddcd172a8d372b9e241ead8f11e1e420769b994cae8b110a34c336f4318" exitCode=0 Oct 30 22:05:07.316258 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.316064 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b8a54f-zq6zv" event={"ID":"ef6aad01-3c06-4cb5-9e33-a43d39a5b917","Type":"ContainerDied","Data":"9ddaeddcd172a8d372b9e241ead8f11e1e420769b994cae8b110a34c336f4318"} Oct 30 22:05:07.317473 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.317450 4560 generic.go:356] "Generic (PLEG): container finished" podID="9e2d7535-075d-4211-b814-35d6abff7605" containerID="464a950cb3eb7661e8c2a30c1b06fd05892a0b474bffc35fdfd16a2bf907abb6" exitCode=0 Oct 30 22:05:07.317551 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.317492 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" event={"ID":"9e2d7535-075d-4211-b814-35d6abff7605","Type":"ContainerDied","Data":"464a950cb3eb7661e8c2a30c1b06fd05892a0b474bffc35fdfd16a2bf907abb6"} Oct 30 22:05:07.317551 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.317498 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" Oct 30 22:05:07.317551 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.317517 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w" event={"ID":"9e2d7535-075d-4211-b814-35d6abff7605","Type":"ContainerDied","Data":"b2c14f67d09e7f4dd951ead410c40532d7e2c8d18112374d9e0fe059057a3cd0"} Oct 30 22:05:07.318607 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.318591 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" Oct 30 22:05:07.320514 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.320491 4560 scope.go:122] "RemoveContainer" containerID="3b09ed402efdf45ad09ae6b962dcff9a0c1711d4707e88e95372afeac16abebc" Oct 30 22:05:07.320777 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:05:07.320750 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b09ed402efdf45ad09ae6b962dcff9a0c1711d4707e88e95372afeac16abebc\": container with ID starting with 3b09ed402efdf45ad09ae6b962dcff9a0c1711d4707e88e95372afeac16abebc not found: ID does not exist" containerID="3b09ed402efdf45ad09ae6b962dcff9a0c1711d4707e88e95372afeac16abebc" Oct 30 22:05:07.320854 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.320787 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b09ed402efdf45ad09ae6b962dcff9a0c1711d4707e88e95372afeac16abebc"} err="failed to get container status \"3b09ed402efdf45ad09ae6b962dcff9a0c1711d4707e88e95372afeac16abebc\": rpc error: code = NotFound desc = could not find container \"3b09ed402efdf45ad09ae6b962dcff9a0c1711d4707e88e95372afeac16abebc\": container with ID starting with 3b09ed402efdf45ad09ae6b962dcff9a0c1711d4707e88e95372afeac16abebc not found: ID does not exist" Oct 30 22:05:07.320854 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.320804 4560 scope.go:122] "RemoveContainer" containerID="464a950cb3eb7661e8c2a30c1b06fd05892a0b474bffc35fdfd16a2bf907abb6" Oct 30 22:05:07.330488 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.330466 4560 scope.go:122] "RemoveContainer" containerID="464a950cb3eb7661e8c2a30c1b06fd05892a0b474bffc35fdfd16a2bf907abb6" Oct 30 22:05:07.330788 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:05:07.330764 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"464a950cb3eb7661e8c2a30c1b06fd05892a0b474bffc35fdfd16a2bf907abb6\": container with ID starting with 464a950cb3eb7661e8c2a30c1b06fd05892a0b474bffc35fdfd16a2bf907abb6 not found: ID does not exist" containerID="464a950cb3eb7661e8c2a30c1b06fd05892a0b474bffc35fdfd16a2bf907abb6" Oct 30 22:05:07.330882 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.330797 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"464a950cb3eb7661e8c2a30c1b06fd05892a0b474bffc35fdfd16a2bf907abb6"} err="failed to get container status \"464a950cb3eb7661e8c2a30c1b06fd05892a0b474bffc35fdfd16a2bf907abb6\": rpc error: code = NotFound desc = could not find container \"464a950cb3eb7661e8c2a30c1b06fd05892a0b474bffc35fdfd16a2bf907abb6\": container with ID starting with 464a950cb3eb7661e8c2a30c1b06fd05892a0b474bffc35fdfd16a2bf907abb6 not found: ID does not exist" Oct 30 22:05:07.350894 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.350872 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90bfb1e4-9526-40f6-aed4-ee46c2d86471-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90bfb1e4-9526-40f6-aed4-ee46c2d86471-tmp-dir\") pod \"90bfb1e4-9526-40f6-aed4-ee46c2d86471\" (UID: \"90bfb1e4-9526-40f6-aed4-ee46c2d86471\") " Oct 30 22:05:07.351018 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.350912 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90bfb1e4-9526-40f6-aed4-ee46c2d86471-home\" (UniqueName: \"kubernetes.io/empty-dir/90bfb1e4-9526-40f6-aed4-ee46c2d86471-home\") pod \"90bfb1e4-9526-40f6-aed4-ee46c2d86471\" (UID: \"90bfb1e4-9526-40f6-aed4-ee46c2d86471\") " Oct 30 22:05:07.351018 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.350940 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90bfb1e4-9526-40f6-aed4-ee46c2d86471-kube-api-access-p9hd6\" (UniqueName: \"kubernetes.io/projected/90bfb1e4-9526-40f6-aed4-ee46c2d86471-kube-api-access-p9hd6\") pod \"90bfb1e4-9526-40f6-aed4-ee46c2d86471\" (UID: \"90bfb1e4-9526-40f6-aed4-ee46c2d86471\") " Oct 30 22:05:07.351018 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.350968 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90bfb1e4-9526-40f6-aed4-ee46c2d86471-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90bfb1e4-9526-40f6-aed4-ee46c2d86471-model-cache\") pod \"90bfb1e4-9526-40f6-aed4-ee46c2d86471\" (UID: \"90bfb1e4-9526-40f6-aed4-ee46c2d86471\") " Oct 30 22:05:07.351018 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.350986 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90bfb1e4-9526-40f6-aed4-ee46c2d86471-dshm\" (UniqueName: \"kubernetes.io/empty-dir/90bfb1e4-9526-40f6-aed4-ee46c2d86471-dshm\") pod \"90bfb1e4-9526-40f6-aed4-ee46c2d86471\" (UID: \"90bfb1e4-9526-40f6-aed4-ee46c2d86471\") " Oct 30 22:05:07.351018 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.351005 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e2d7535-075d-4211-b814-35d6abff7605-kube-api-access-nbnml\" (UniqueName: \"kubernetes.io/projected/9e2d7535-075d-4211-b814-35d6abff7605-kube-api-access-nbnml\") pod \"9e2d7535-075d-4211-b814-35d6abff7605\" (UID: \"9e2d7535-075d-4211-b814-35d6abff7605\") " Oct 30 22:05:07.351313 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.351040 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-tls-certs\" (UniqueName: \"kubernetes.io/secret/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-tls-certs\") pod \"971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d\" (UID: \"971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d\") " Oct 30 22:05:07.351313 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.351086 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a59a0fc3-8316-47eb-a42e-552d202b1ab7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a59a0fc3-8316-47eb-a42e-552d202b1ab7-model-cache\") pod \"a59a0fc3-8316-47eb-a42e-552d202b1ab7\" (UID: \"a59a0fc3-8316-47eb-a42e-552d202b1ab7\") " Oct 30 22:05:07.351313 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.351114 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-model-cache\") pod \"971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d\" (UID: \"971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d\") " Oct 30 22:05:07.351313 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.351161 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e2d7535-075d-4211-b814-35d6abff7605-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e2d7535-075d-4211-b814-35d6abff7605-tls-certs\") pod \"9e2d7535-075d-4211-b814-35d6abff7605\" (UID: \"9e2d7535-075d-4211-b814-35d6abff7605\") " Oct 30 22:05:07.351313 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.351191 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a59a0fc3-8316-47eb-a42e-552d202b1ab7-home\" (UniqueName: \"kubernetes.io/empty-dir/a59a0fc3-8316-47eb-a42e-552d202b1ab7-home\") pod \"a59a0fc3-8316-47eb-a42e-552d202b1ab7\" (UID: \"a59a0fc3-8316-47eb-a42e-552d202b1ab7\") " Oct 30 22:05:07.351313 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.351197 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90bfb1e4-9526-40f6-aed4-ee46c2d86471-home" pod "90bfb1e4-9526-40f6-aed4-ee46c2d86471" (UID: "90bfb1e4-9526-40f6-aed4-ee46c2d86471"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:07.351313 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.351231 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-dshm\") pod \"971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d\" (UID: \"971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d\") " Oct 30 22:05:07.351313 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.351222 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90bfb1e4-9526-40f6-aed4-ee46c2d86471-tmp-dir" pod "90bfb1e4-9526-40f6-aed4-ee46c2d86471" (UID: "90bfb1e4-9526-40f6-aed4-ee46c2d86471"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:07.351313 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.351270 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90bfb1e4-9526-40f6-aed4-ee46c2d86471-model-cache" pod "90bfb1e4-9526-40f6-aed4-ee46c2d86471" (UID: "90bfb1e4-9526-40f6-aed4-ee46c2d86471"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:07.351313 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.351262 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e2d7535-075d-4211-b814-35d6abff7605-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e2d7535-075d-4211-b814-35d6abff7605-tmp-dir\") pod \"9e2d7535-075d-4211-b814-35d6abff7605\" (UID: \"9e2d7535-075d-4211-b814-35d6abff7605\") " Oct 30 22:05:07.351859 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.351317 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e2d7535-075d-4211-b814-35d6abff7605-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e2d7535-075d-4211-b814-35d6abff7605-model-cache\") pod \"9e2d7535-075d-4211-b814-35d6abff7605\" (UID: \"9e2d7535-075d-4211-b814-35d6abff7605\") " Oct 30 22:05:07.351859 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.351388 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90bfb1e4-9526-40f6-aed4-ee46c2d86471-tls-certs\" (UniqueName: \"kubernetes.io/secret/90bfb1e4-9526-40f6-aed4-ee46c2d86471-tls-certs\") pod \"90bfb1e4-9526-40f6-aed4-ee46c2d86471\" (UID: \"90bfb1e4-9526-40f6-aed4-ee46c2d86471\") " Oct 30 22:05:07.351859 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.351434 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-tmp-dir\") pod \"971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d\" (UID: \"971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d\") " Oct 30 22:05:07.351859 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.351465 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2d7535-075d-4211-b814-35d6abff7605-tmp-dir" pod "9e2d7535-075d-4211-b814-35d6abff7605" (UID: "9e2d7535-075d-4211-b814-35d6abff7605"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:07.351859 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.351476 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a59a0fc3-8316-47eb-a42e-552d202b1ab7-kube-api-access-cgqnf\" (UniqueName: \"kubernetes.io/projected/a59a0fc3-8316-47eb-a42e-552d202b1ab7-kube-api-access-cgqnf\") pod \"a59a0fc3-8316-47eb-a42e-552d202b1ab7\" (UID: \"a59a0fc3-8316-47eb-a42e-552d202b1ab7\") " Oct 30 22:05:07.351859 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.351479 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-model-cache" pod "971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d" (UID: "971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:07.351859 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.351482 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a59a0fc3-8316-47eb-a42e-552d202b1ab7-home" pod "a59a0fc3-8316-47eb-a42e-552d202b1ab7" (UID: "a59a0fc3-8316-47eb-a42e-552d202b1ab7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:07.351859 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.351571 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a59a0fc3-8316-47eb-a42e-552d202b1ab7-tls-certs\" (UniqueName: \"kubernetes.io/secret/a59a0fc3-8316-47eb-a42e-552d202b1ab7-tls-certs\") pod \"a59a0fc3-8316-47eb-a42e-552d202b1ab7\" (UID: \"a59a0fc3-8316-47eb-a42e-552d202b1ab7\") " Oct 30 22:05:07.351859 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.351606 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e2d7535-075d-4211-b814-35d6abff7605-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e2d7535-075d-4211-b814-35d6abff7605-dshm\") pod \"9e2d7535-075d-4211-b814-35d6abff7605\" (UID: \"9e2d7535-075d-4211-b814-35d6abff7605\") " Oct 30 22:05:07.351859 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.351650 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-home\" (UniqueName: \"kubernetes.io/empty-dir/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-home\") pod \"971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d\" (UID: \"971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d\") " Oct 30 22:05:07.351859 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.351684 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e2d7535-075d-4211-b814-35d6abff7605-home\" (UniqueName: \"kubernetes.io/empty-dir/9e2d7535-075d-4211-b814-35d6abff7605-home\") pod \"9e2d7535-075d-4211-b814-35d6abff7605\" (UID: \"9e2d7535-075d-4211-b814-35d6abff7605\") " Oct 30 22:05:07.351859 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.351724 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-kube-api-access-5452f\" (UniqueName: \"kubernetes.io/projected/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-kube-api-access-5452f\") pod \"971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d\" (UID: \"971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d\") " Oct 30 22:05:07.351859 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.351754 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a59a0fc3-8316-47eb-a42e-552d202b1ab7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a59a0fc3-8316-47eb-a42e-552d202b1ab7-tmp-dir\") pod \"a59a0fc3-8316-47eb-a42e-552d202b1ab7\" (UID: \"a59a0fc3-8316-47eb-a42e-552d202b1ab7\") " Oct 30 22:05:07.351859 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.351788 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a59a0fc3-8316-47eb-a42e-552d202b1ab7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a59a0fc3-8316-47eb-a42e-552d202b1ab7-dshm\") pod \"a59a0fc3-8316-47eb-a42e-552d202b1ab7\" (UID: \"a59a0fc3-8316-47eb-a42e-552d202b1ab7\") " Oct 30 22:05:07.352632 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.352143 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90bfb1e4-9526-40f6-aed4-ee46c2d86471-home\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:07.352632 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.352162 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90bfb1e4-9526-40f6-aed4-ee46c2d86471-model-cache\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:07.352632 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.352175 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-model-cache\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:07.352632 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.352193 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a59a0fc3-8316-47eb-a42e-552d202b1ab7-home\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:07.352632 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.352207 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e2d7535-075d-4211-b814-35d6abff7605-tmp-dir\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:07.352632 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.352220 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90bfb1e4-9526-40f6-aed4-ee46c2d86471-tmp-dir\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:07.354073 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.353906 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90bfb1e4-9526-40f6-aed4-ee46c2d86471-kube-api-access-p9hd6" pod "90bfb1e4-9526-40f6-aed4-ee46c2d86471" (UID: "90bfb1e4-9526-40f6-aed4-ee46c2d86471"). InnerVolumeSpecName "kube-api-access-p9hd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:07.354073 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.353927 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2d7535-075d-4211-b814-35d6abff7605-model-cache" pod "9e2d7535-075d-4211-b814-35d6abff7605" (UID: "9e2d7535-075d-4211-b814-35d6abff7605"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:07.354073 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.353928 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-home" pod "971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d" (UID: "971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:07.354073 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.353990 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2d7535-075d-4211-b814-35d6abff7605-home" pod "9e2d7535-075d-4211-b814-35d6abff7605" (UID: "9e2d7535-075d-4211-b814-35d6abff7605"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:07.354073 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.353994 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a59a0fc3-8316-47eb-a42e-552d202b1ab7-model-cache" pod "a59a0fc3-8316-47eb-a42e-552d202b1ab7" (UID: "a59a0fc3-8316-47eb-a42e-552d202b1ab7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:07.354426 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.354116 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-tmp-dir" pod "971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d" (UID: "971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:07.354500 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.354454 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-dshm" pod "971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d" (UID: "971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:07.354500 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.354472 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90bfb1e4-9526-40f6-aed4-ee46c2d86471-dshm" pod "90bfb1e4-9526-40f6-aed4-ee46c2d86471" (UID: "90bfb1e4-9526-40f6-aed4-ee46c2d86471"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:07.354620 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.354600 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a59a0fc3-8316-47eb-a42e-552d202b1ab7-dshm" pod "a59a0fc3-8316-47eb-a42e-552d202b1ab7" (UID: "a59a0fc3-8316-47eb-a42e-552d202b1ab7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:07.354726 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.354692 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a59a0fc3-8316-47eb-a42e-552d202b1ab7-tmp-dir" pod "a59a0fc3-8316-47eb-a42e-552d202b1ab7" (UID: "a59a0fc3-8316-47eb-a42e-552d202b1ab7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:07.354844 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.354818 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90bfb1e4-9526-40f6-aed4-ee46c2d86471-tls-certs" pod "90bfb1e4-9526-40f6-aed4-ee46c2d86471" (UID: "90bfb1e4-9526-40f6-aed4-ee46c2d86471"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:07.355094 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.355066 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-tls-certs" pod "971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d" (UID: "971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:07.355426 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.355389 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e2d7535-075d-4211-b814-35d6abff7605-tls-certs" pod "9e2d7535-075d-4211-b814-35d6abff7605" (UID: "9e2d7535-075d-4211-b814-35d6abff7605"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:07.355532 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.355495 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e2d7535-075d-4211-b814-35d6abff7605-kube-api-access-nbnml" pod "9e2d7535-075d-4211-b814-35d6abff7605" (UID: "9e2d7535-075d-4211-b814-35d6abff7605"). InnerVolumeSpecName "kube-api-access-nbnml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:07.355612 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.355594 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2d7535-075d-4211-b814-35d6abff7605-dshm" pod "9e2d7535-075d-4211-b814-35d6abff7605" (UID: "9e2d7535-075d-4211-b814-35d6abff7605"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:07.356152 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.356126 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-kube-api-access-5452f" pod "971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d" (UID: "971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d"). InnerVolumeSpecName "kube-api-access-5452f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:07.356232 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.356183 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a59a0fc3-8316-47eb-a42e-552d202b1ab7-kube-api-access-cgqnf" pod "a59a0fc3-8316-47eb-a42e-552d202b1ab7" (UID: "a59a0fc3-8316-47eb-a42e-552d202b1ab7"). InnerVolumeSpecName "kube-api-access-cgqnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:07.356369 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.356321 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a59a0fc3-8316-47eb-a42e-552d202b1ab7-tls-certs" pod "a59a0fc3-8316-47eb-a42e-552d202b1ab7" (UID: "a59a0fc3-8316-47eb-a42e-552d202b1ab7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:07.453064 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.452983 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e2d7535-075d-4211-b814-35d6abff7605-model-cache\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:07.453064 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.453012 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90bfb1e4-9526-40f6-aed4-ee46c2d86471-tls-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:07.453064 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.453022 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-tmp-dir\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:07.453064 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.453032 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgqnf\" (UniqueName: \"kubernetes.io/projected/a59a0fc3-8316-47eb-a42e-552d202b1ab7-kube-api-access-cgqnf\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:07.453064 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.453042 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a59a0fc3-8316-47eb-a42e-552d202b1ab7-tls-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:07.453064 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.453051 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e2d7535-075d-4211-b814-35d6abff7605-dshm\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:07.453064 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.453058 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-home\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:07.453064 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.453065 4560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e2d7535-075d-4211-b814-35d6abff7605-home\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:07.453449 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.453074 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5452f\" (UniqueName: \"kubernetes.io/projected/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-kube-api-access-5452f\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:07.453449 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.453083 4560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a59a0fc3-8316-47eb-a42e-552d202b1ab7-tmp-dir\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:07.453449 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.453091 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a59a0fc3-8316-47eb-a42e-552d202b1ab7-dshm\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:07.453449 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.453098 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9hd6\" (UniqueName: \"kubernetes.io/projected/90bfb1e4-9526-40f6-aed4-ee46c2d86471-kube-api-access-p9hd6\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:07.453449 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.453107 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90bfb1e4-9526-40f6-aed4-ee46c2d86471-dshm\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:07.453449 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.453115 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbnml\" (UniqueName: \"kubernetes.io/projected/9e2d7535-075d-4211-b814-35d6abff7605-kube-api-access-nbnml\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:07.453449 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.453123 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-tls-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:07.453449 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.453131 4560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a59a0fc3-8316-47eb-a42e-552d202b1ab7-model-cache\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:07.453449 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.453138 4560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e2d7535-075d-4211-b814-35d6abff7605-tls-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:07.453449 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.453146 4560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d-dshm\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:07.973395 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.973356 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w"] Oct 30 22:05:07.976093 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:07.976069 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-afbe5b/distinct-2-w0-afbe5b-kserve-7fb4cd88d7-rsf7w"] Oct 30 22:05:08.079693 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:08.079656 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd"] Oct 30 22:05:08.083369 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:08.083333 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-afbe5b/distinct-w0-afbe5b-kserve-9f845d77b-kftqd"] Oct 30 22:05:08.326487 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:08.326457 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" Oct 30 22:05:08.326956 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:08.326501 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx" event={"ID":"90bfb1e4-9526-40f6-aed4-ee46c2d86471","Type":"ContainerDied","Data":"9d6436214a40f8f2fde67686c4fa6861e700314abb6e0ba4575c43d8f5fa3830"} Oct 30 22:05:08.326956 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:08.326547 4560 scope.go:122] "RemoveContainer" containerID="83471276efe32bda9d7935fc7b28bf80bc1c1a901efd72e572720261129c37bb" Oct 30 22:05:08.327917 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:08.327762 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" Oct 30 22:05:08.327917 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:08.327781 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9" event={"ID":"a59a0fc3-8316-47eb-a42e-552d202b1ab7","Type":"ContainerDied","Data":"e08b506f3f7481b6489a9d788ac7c848e4bb23605224d74da0380c309730f764"} Oct 30 22:05:08.339006 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:08.338876 4560 scope.go:122] "RemoveContainer" containerID="3d9ac6e9d0d2264fe8c1ecfa3b11164581832dce26b6b645f05d81fb438adca5" Oct 30 22:05:08.467394 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:08.467370 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b8a54f-zq6zv" Oct 30 22:05:08.481728 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:08.481700 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9"] Oct 30 22:05:08.485994 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:08.485971 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-afbe5b/premium-w0-afbe5b-kserve-5c568776bf-6wqw9"] Oct 30 22:05:08.566180 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:08.566121 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef6aad01-3c06-4cb5-9e33-a43d39a5b917-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef6aad01-3c06-4cb5-9e33-a43d39a5b917-maas-api-openshift-ca\") pod \"ef6aad01-3c06-4cb5-9e33-a43d39a5b917\" (UID: \"ef6aad01-3c06-4cb5-9e33-a43d39a5b917\") " Oct 30 22:05:08.566406 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:08.566194 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef6aad01-3c06-4cb5-9e33-a43d39a5b917-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef6aad01-3c06-4cb5-9e33-a43d39a5b917-maas-api-xks-ca\") pod \"ef6aad01-3c06-4cb5-9e33-a43d39a5b917\" (UID: \"ef6aad01-3c06-4cb5-9e33-a43d39a5b917\") " Oct 30 22:05:08.566406 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:08.566311 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef6aad01-3c06-4cb5-9e33-a43d39a5b917-kube-api-access-7n7pw\" (UniqueName: \"kubernetes.io/projected/ef6aad01-3c06-4cb5-9e33-a43d39a5b917-kube-api-access-7n7pw\") pod \"ef6aad01-3c06-4cb5-9e33-a43d39a5b917\" (UID: \"ef6aad01-3c06-4cb5-9e33-a43d39a5b917\") " Oct 30 22:05:08.566582 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:08.566561 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef6aad01-3c06-4cb5-9e33-a43d39a5b917-maas-api-openshift-ca" pod "ef6aad01-3c06-4cb5-9e33-a43d39a5b917" (UID: "ef6aad01-3c06-4cb5-9e33-a43d39a5b917"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:08.566755 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:08.566668 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef6aad01-3c06-4cb5-9e33-a43d39a5b917-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:08.568445 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:08.568419 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef6aad01-3c06-4cb5-9e33-a43d39a5b917-maas-api-xks-ca" pod "ef6aad01-3c06-4cb5-9e33-a43d39a5b917" (UID: "ef6aad01-3c06-4cb5-9e33-a43d39a5b917"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:08.568445 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:08.568444 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef6aad01-3c06-4cb5-9e33-a43d39a5b917-kube-api-access-7n7pw" pod "ef6aad01-3c06-4cb5-9e33-a43d39a5b917" (UID: "ef6aad01-3c06-4cb5-9e33-a43d39a5b917"). InnerVolumeSpecName "kube-api-access-7n7pw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:08.574731 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:08.574706 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d" path="/var/lib/kubelet/pods/971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d/volumes" Oct 30 22:05:08.575158 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:08.575140 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e2d7535-075d-4211-b814-35d6abff7605" path="/var/lib/kubelet/pods/9e2d7535-075d-4211-b814-35d6abff7605/volumes" Oct 30 22:05:08.575584 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:08.575566 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a59a0fc3-8316-47eb-a42e-552d202b1ab7" path="/var/lib/kubelet/pods/a59a0fc3-8316-47eb-a42e-552d202b1ab7/volumes" Oct 30 22:05:08.589098 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:08.589073 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx"] Oct 30 22:05:08.593843 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:08.593819 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-afbe5b/simulator-w0-afbe5b-kserve-8dd785bfb-vn7kx"] Oct 30 22:05:08.667917 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:08.667884 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n7pw\" (UniqueName: \"kubernetes.io/projected/ef6aad01-3c06-4cb5-9e33-a43d39a5b917-kube-api-access-7n7pw\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:08.667917 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:08.667913 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef6aad01-3c06-4cb5-9e33-a43d39a5b917-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:09.164646 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:09.164597 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-afbe5b-85d55db988-8c4wd" podUID="024e6f83-289f-499a-be2d-5aed8a5a6c6d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:05:09.164829 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:09.164615 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-afbe5b-646894cd95-h5tqj" podUID="1807c151-3102-4158-b028-d7dd697f4ad6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:05:09.336194 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:09.336164 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b8a54f-zq6zv" Oct 30 22:05:09.336610 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:09.336195 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b8a54f-zq6zv" event={"ID":"ef6aad01-3c06-4cb5-9e33-a43d39a5b917","Type":"ContainerDied","Data":"720885257a9497eadf255a8a6a8ce6942491434b910a64d934365adf0b893cac"} Oct 30 22:05:09.336610 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:09.336265 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="720885257a9497eadf255a8a6a8ce6942491434b910a64d934365adf0b893cac" Oct 30 22:05:10.574672 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:10.574640 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90bfb1e4-9526-40f6-aed4-ee46c2d86471" path="/var/lib/kubelet/pods/90bfb1e4-9526-40f6-aed4-ee46c2d86471/volumes" Oct 30 22:05:19.164941 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:19.164900 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-afbe5b-646894cd95-h5tqj" podUID="1807c151-3102-4158-b028-d7dd697f4ad6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:05:19.165317 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:19.164918 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-afbe5b-85d55db988-8c4wd" podUID="024e6f83-289f-499a-be2d-5aed8a5a6c6d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:05:19.594734 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:19.594698 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:05:19.683429 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:05:19.683392 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:05:22.829990 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:22.829957 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-afbe5b-qpvcj"] Oct 30 22:05:22.877162 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:22.877122 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-afbe5b-qpvcj"] Oct 30 22:05:24.421384 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:24.421352 4560 generic.go:356] "Generic (PLEG): container finished" podID="024e6f83-289f-499a-be2d-5aed8a5a6c6d" containerID="2660ad319e7c9efa9255615da5d042b7aa05f14a25ba1091e2ae4464ba32134a" exitCode=0 Oct 30 22:05:24.421739 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:24.421380 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-afbe5b-85d55db988-8c4wd" event={"ID":"024e6f83-289f-499a-be2d-5aed8a5a6c6d","Type":"ContainerDied","Data":"2660ad319e7c9efa9255615da5d042b7aa05f14a25ba1091e2ae4464ba32134a"} Oct 30 22:05:24.492092 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:24.492066 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-afbe5b-85d55db988-8c4wd" Oct 30 22:05:24.574458 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:24.574429 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23ddc9db-728d-4e3c-a147-0eb3e070ad43" path="/var/lib/kubelet/pods/23ddc9db-728d-4e3c-a147-0eb3e070ad43/volumes" Oct 30 22:05:24.590422 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:24.590399 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-afbe5b-646894cd95-h5tqj" Oct 30 22:05:24.623074 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:24.623048 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/024e6f83-289f-499a-be2d-5aed8a5a6c6d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/024e6f83-289f-499a-be2d-5aed8a5a6c6d-plugins-config-volume\") pod \"024e6f83-289f-499a-be2d-5aed8a5a6c6d\" (UID: \"024e6f83-289f-499a-be2d-5aed8a5a6c6d\") " Oct 30 22:05:24.623211 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:24.623105 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/024e6f83-289f-499a-be2d-5aed8a5a6c6d-kube-api-access-57t2t\" (UniqueName: \"kubernetes.io/projected/024e6f83-289f-499a-be2d-5aed8a5a6c6d-kube-api-access-57t2t\") pod \"024e6f83-289f-499a-be2d-5aed8a5a6c6d\" (UID: \"024e6f83-289f-499a-be2d-5aed8a5a6c6d\") " Oct 30 22:05:24.623547 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:24.623522 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/024e6f83-289f-499a-be2d-5aed8a5a6c6d-plugins-config-volume" pod "024e6f83-289f-499a-be2d-5aed8a5a6c6d" (UID: "024e6f83-289f-499a-be2d-5aed8a5a6c6d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:24.625165 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:24.625144 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/024e6f83-289f-499a-be2d-5aed8a5a6c6d-kube-api-access-57t2t" pod "024e6f83-289f-499a-be2d-5aed8a5a6c6d" (UID: "024e6f83-289f-499a-be2d-5aed8a5a6c6d"). InnerVolumeSpecName "kube-api-access-57t2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:24.724015 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:24.723982 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1807c151-3102-4158-b028-d7dd697f4ad6-kube-api-access-7rdr5\" (UniqueName: \"kubernetes.io/projected/1807c151-3102-4158-b028-d7dd697f4ad6-kube-api-access-7rdr5\") pod \"1807c151-3102-4158-b028-d7dd697f4ad6\" (UID: \"1807c151-3102-4158-b028-d7dd697f4ad6\") " Oct 30 22:05:24.724187 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:24.724051 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1807c151-3102-4158-b028-d7dd697f4ad6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1807c151-3102-4158-b028-d7dd697f4ad6-plugins-config-volume\") pod \"1807c151-3102-4158-b028-d7dd697f4ad6\" (UID: \"1807c151-3102-4158-b028-d7dd697f4ad6\") " Oct 30 22:05:24.724363 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:24.724323 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/024e6f83-289f-499a-be2d-5aed8a5a6c6d-plugins-config-volume\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:24.724445 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:24.724370 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57t2t\" (UniqueName: \"kubernetes.io/projected/024e6f83-289f-499a-be2d-5aed8a5a6c6d-kube-api-access-57t2t\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:24.724445 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:24.724420 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1807c151-3102-4158-b028-d7dd697f4ad6-plugins-config-volume" pod "1807c151-3102-4158-b028-d7dd697f4ad6" (UID: "1807c151-3102-4158-b028-d7dd697f4ad6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:24.726259 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:24.726232 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1807c151-3102-4158-b028-d7dd697f4ad6-kube-api-access-7rdr5" pod "1807c151-3102-4158-b028-d7dd697f4ad6" (UID: "1807c151-3102-4158-b028-d7dd697f4ad6"). InnerVolumeSpecName "kube-api-access-7rdr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:24.824879 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:24.824792 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rdr5\" (UniqueName: \"kubernetes.io/projected/1807c151-3102-4158-b028-d7dd697f4ad6-kube-api-access-7rdr5\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:24.824879 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:24.824824 4560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1807c151-3102-4158-b028-d7dd697f4ad6-plugins-config-volume\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:25.428021 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:25.427979 4560 generic.go:356] "Generic (PLEG): container finished" podID="1807c151-3102-4158-b028-d7dd697f4ad6" containerID="6571f2bf6dceb7f69dac253f0b49495f4b43dd0f9035ebec675e8bed5206716f" exitCode=0 Oct 30 22:05:25.428563 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:25.428041 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-afbe5b-646894cd95-h5tqj" Oct 30 22:05:25.428563 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:25.428065 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-afbe5b-646894cd95-h5tqj" event={"ID":"1807c151-3102-4158-b028-d7dd697f4ad6","Type":"ContainerDied","Data":"6571f2bf6dceb7f69dac253f0b49495f4b43dd0f9035ebec675e8bed5206716f"} Oct 30 22:05:25.428563 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:25.428173 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-afbe5b-646894cd95-h5tqj" event={"ID":"1807c151-3102-4158-b028-d7dd697f4ad6","Type":"ContainerDied","Data":"14a381763658f8b3da22bfc2e935840002b561a880c6bacb4e763832f25a10a8"} Oct 30 22:05:25.428563 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:25.428209 4560 scope.go:122] "RemoveContainer" containerID="6571f2bf6dceb7f69dac253f0b49495f4b43dd0f9035ebec675e8bed5206716f" Oct 30 22:05:25.429660 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:25.429645 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-afbe5b-85d55db988-8c4wd" Oct 30 22:05:25.429744 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:25.429677 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-afbe5b-85d55db988-8c4wd" event={"ID":"024e6f83-289f-499a-be2d-5aed8a5a6c6d","Type":"ContainerDied","Data":"5af9476a3d41ed91cdd79a4fc681c9362bd7bfe6b5c619ad0404dbea0e089197"} Oct 30 22:05:25.438942 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:25.438924 4560 scope.go:122] "RemoveContainer" containerID="6571f2bf6dceb7f69dac253f0b49495f4b43dd0f9035ebec675e8bed5206716f" Oct 30 22:05:25.439217 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:05:25.439199 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6571f2bf6dceb7f69dac253f0b49495f4b43dd0f9035ebec675e8bed5206716f\": container with ID starting with 6571f2bf6dceb7f69dac253f0b49495f4b43dd0f9035ebec675e8bed5206716f not found: ID does not exist" containerID="6571f2bf6dceb7f69dac253f0b49495f4b43dd0f9035ebec675e8bed5206716f" Oct 30 22:05:25.439279 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:25.439225 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6571f2bf6dceb7f69dac253f0b49495f4b43dd0f9035ebec675e8bed5206716f"} err="failed to get container status \"6571f2bf6dceb7f69dac253f0b49495f4b43dd0f9035ebec675e8bed5206716f\": rpc error: code = NotFound desc = could not find container \"6571f2bf6dceb7f69dac253f0b49495f4b43dd0f9035ebec675e8bed5206716f\": container with ID starting with 6571f2bf6dceb7f69dac253f0b49495f4b43dd0f9035ebec675e8bed5206716f not found: ID does not exist" Oct 30 22:05:25.439279 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:25.439241 4560 scope.go:122] "RemoveContainer" containerID="2660ad319e7c9efa9255615da5d042b7aa05f14a25ba1091e2ae4464ba32134a" Oct 30 22:05:25.583886 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:25.583809 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-afbe5b-646894cd95-h5tqj"] Oct 30 22:05:25.630799 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:25.630767 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-afbe5b-646894cd95-h5tqj"] Oct 30 22:05:25.735073 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:25.735041 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-afbe5b-85d55db988-8c4wd"] Oct 30 22:05:25.741841 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:25.741816 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-afbe5b-85d55db988-8c4wd"] Oct 30 22:05:26.573945 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:26.573914 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="024e6f83-289f-499a-be2d-5aed8a5a6c6d" path="/var/lib/kubelet/pods/024e6f83-289f-499a-be2d-5aed8a5a6c6d/volumes" Oct 30 22:05:26.574402 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:26.574263 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1807c151-3102-4158-b028-d7dd697f4ad6" path="/var/lib/kubelet/pods/1807c151-3102-4158-b028-d7dd697f4ad6/volumes" Oct 30 22:05:31.803754 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:31.803722 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp"] Oct 30 22:05:31.804294 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:31.803986 4560 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" podUID="f10eae6b-051d-4929-902f-abcd38146a15" containerName="istio-proxy" containerID="cri-o://61b1b054f2ab58b12b0dc2fb000f420582a975311487be609addeeea4e488779" gracePeriod=30 Oct 30 22:05:35.265597 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:35.265559 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:05:37.046683 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:37.046656 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" Oct 30 22:05:37.142325 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:37.142247 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f10eae6b-051d-4929-902f-abcd38146a15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f10eae6b-051d-4929-902f-abcd38146a15-istiod-ca-cert\") pod \"f10eae6b-051d-4929-902f-abcd38146a15\" (UID: \"f10eae6b-051d-4929-902f-abcd38146a15\") " Oct 30 22:05:37.142325 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:37.142296 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f10eae6b-051d-4929-902f-abcd38146a15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f10eae6b-051d-4929-902f-abcd38146a15-istio-envoy\") pod \"f10eae6b-051d-4929-902f-abcd38146a15\" (UID: \"f10eae6b-051d-4929-902f-abcd38146a15\") " Oct 30 22:05:37.142325 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:37.142320 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f10eae6b-051d-4929-902f-abcd38146a15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f10eae6b-051d-4929-902f-abcd38146a15-istio-data\") pod \"f10eae6b-051d-4929-902f-abcd38146a15\" (UID: \"f10eae6b-051d-4929-902f-abcd38146a15\") " Oct 30 22:05:37.142631 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:37.142366 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f10eae6b-051d-4929-902f-abcd38146a15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10eae6b-051d-4929-902f-abcd38146a15-workload-socket\") pod \"f10eae6b-051d-4929-902f-abcd38146a15\" (UID: \"f10eae6b-051d-4929-902f-abcd38146a15\") " Oct 30 22:05:37.142631 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:37.142416 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f10eae6b-051d-4929-902f-abcd38146a15-istio-token\" (UniqueName: \"kubernetes.io/projected/f10eae6b-051d-4929-902f-abcd38146a15-istio-token\") pod \"f10eae6b-051d-4929-902f-abcd38146a15\" (UID: \"f10eae6b-051d-4929-902f-abcd38146a15\") " Oct 30 22:05:37.142631 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:37.142465 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f10eae6b-051d-4929-902f-abcd38146a15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f10eae6b-051d-4929-902f-abcd38146a15-istio-podinfo\") pod \"f10eae6b-051d-4929-902f-abcd38146a15\" (UID: \"f10eae6b-051d-4929-902f-abcd38146a15\") " Oct 30 22:05:37.142631 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:37.142519 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f10eae6b-051d-4929-902f-abcd38146a15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f10eae6b-051d-4929-902f-abcd38146a15-workload-certs\") pod \"f10eae6b-051d-4929-902f-abcd38146a15\" (UID: \"f10eae6b-051d-4929-902f-abcd38146a15\") " Oct 30 22:05:37.142631 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:37.142576 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f10eae6b-051d-4929-902f-abcd38146a15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10eae6b-051d-4929-902f-abcd38146a15-credential-socket\") pod \"f10eae6b-051d-4929-902f-abcd38146a15\" (UID: \"f10eae6b-051d-4929-902f-abcd38146a15\") " Oct 30 22:05:37.142631 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:37.142607 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f10eae6b-051d-4929-902f-abcd38146a15-kube-api-access-9plzk\" (UniqueName: \"kubernetes.io/projected/f10eae6b-051d-4929-902f-abcd38146a15-kube-api-access-9plzk\") pod \"f10eae6b-051d-4929-902f-abcd38146a15\" (UID: \"f10eae6b-051d-4929-902f-abcd38146a15\") " Oct 30 22:05:37.142972 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:37.142688 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10eae6b-051d-4929-902f-abcd38146a15-istio-data" pod "f10eae6b-051d-4929-902f-abcd38146a15" (UID: "f10eae6b-051d-4929-902f-abcd38146a15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:37.142972 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:37.142838 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f10eae6b-051d-4929-902f-abcd38146a15-istiod-ca-cert" pod "f10eae6b-051d-4929-902f-abcd38146a15" (UID: "f10eae6b-051d-4929-902f-abcd38146a15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:37.142972 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:37.142868 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10eae6b-051d-4929-902f-abcd38146a15-workload-certs" pod "f10eae6b-051d-4929-902f-abcd38146a15" (UID: "f10eae6b-051d-4929-902f-abcd38146a15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:37.142972 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:37.142897 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10eae6b-051d-4929-902f-abcd38146a15-credential-socket" pod "f10eae6b-051d-4929-902f-abcd38146a15" (UID: "f10eae6b-051d-4929-902f-abcd38146a15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:37.143202 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:37.142988 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10eae6b-051d-4929-902f-abcd38146a15-workload-socket" pod "f10eae6b-051d-4929-902f-abcd38146a15" (UID: "f10eae6b-051d-4929-902f-abcd38146a15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:37.143202 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:37.143047 4560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10eae6b-051d-4929-902f-abcd38146a15-credential-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:37.143202 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:37.143070 4560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f10eae6b-051d-4929-902f-abcd38146a15-istiod-ca-cert\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:37.143202 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:37.143085 4560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f10eae6b-051d-4929-902f-abcd38146a15-istio-data\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:37.143202 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:37.143098 4560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f10eae6b-051d-4929-902f-abcd38146a15-workload-certs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:37.144717 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:37.144681 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10eae6b-051d-4929-902f-abcd38146a15-istio-envoy" pod "f10eae6b-051d-4929-902f-abcd38146a15" (UID: "f10eae6b-051d-4929-902f-abcd38146a15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:37.144998 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:37.144966 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f10eae6b-051d-4929-902f-abcd38146a15-kube-api-access-9plzk" pod "f10eae6b-051d-4929-902f-abcd38146a15" (UID: "f10eae6b-051d-4929-902f-abcd38146a15"). InnerVolumeSpecName "kube-api-access-9plzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:37.145116 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:37.145096 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f10eae6b-051d-4929-902f-abcd38146a15-istio-podinfo" pod "f10eae6b-051d-4929-902f-abcd38146a15" (UID: "f10eae6b-051d-4929-902f-abcd38146a15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:05:37.145180 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:37.145120 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f10eae6b-051d-4929-902f-abcd38146a15-istio-token" pod "f10eae6b-051d-4929-902f-abcd38146a15" (UID: "f10eae6b-051d-4929-902f-abcd38146a15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:37.243639 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:37.243605 4560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f10eae6b-051d-4929-902f-abcd38146a15-istio-podinfo\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:37.243639 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:37.243638 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9plzk\" (UniqueName: \"kubernetes.io/projected/f10eae6b-051d-4929-902f-abcd38146a15-kube-api-access-9plzk\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:37.243881 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:37.243654 4560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f10eae6b-051d-4929-902f-abcd38146a15-istio-envoy\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:37.243881 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:37.243667 4560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10eae6b-051d-4929-902f-abcd38146a15-workload-socket\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:37.243881 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:37.243679 4560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f10eae6b-051d-4929-902f-abcd38146a15-istio-token\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:37.497846 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:37.497810 4560 generic.go:356] "Generic (PLEG): container finished" podID="f10eae6b-051d-4929-902f-abcd38146a15" containerID="61b1b054f2ab58b12b0dc2fb000f420582a975311487be609addeeea4e488779" exitCode=0 Oct 30 22:05:37.498027 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:37.497872 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" Oct 30 22:05:37.498027 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:37.497899 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" event={"ID":"f10eae6b-051d-4929-902f-abcd38146a15","Type":"ContainerDied","Data":"61b1b054f2ab58b12b0dc2fb000f420582a975311487be609addeeea4e488779"} Oct 30 22:05:37.498027 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:37.497982 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp" event={"ID":"f10eae6b-051d-4929-902f-abcd38146a15","Type":"ContainerDied","Data":"69200a2eb9b7a974012168612839f9f243478da9e8ef21aab40d279da069407c"} Oct 30 22:05:37.498027 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:37.498004 4560 scope.go:122] "RemoveContainer" containerID="61b1b054f2ab58b12b0dc2fb000f420582a975311487be609addeeea4e488779" Oct 30 22:05:37.508233 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:37.508219 4560 scope.go:122] "RemoveContainer" containerID="61b1b054f2ab58b12b0dc2fb000f420582a975311487be609addeeea4e488779" Oct 30 22:05:37.508520 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:05:37.508501 4560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61b1b054f2ab58b12b0dc2fb000f420582a975311487be609addeeea4e488779\": container with ID starting with 61b1b054f2ab58b12b0dc2fb000f420582a975311487be609addeeea4e488779 not found: ID does not exist" containerID="61b1b054f2ab58b12b0dc2fb000f420582a975311487be609addeeea4e488779" Oct 30 22:05:37.508577 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:37.508530 4560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61b1b054f2ab58b12b0dc2fb000f420582a975311487be609addeeea4e488779"} err="failed to get container status \"61b1b054f2ab58b12b0dc2fb000f420582a975311487be609addeeea4e488779\": rpc error: code = NotFound desc = could not find container \"61b1b054f2ab58b12b0dc2fb000f420582a975311487be609addeeea4e488779\": container with ID starting with 61b1b054f2ab58b12b0dc2fb000f420582a975311487be609addeeea4e488779 not found: ID does not exist" Oct 30 22:05:37.653423 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:37.653390 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp"] Oct 30 22:05:37.699938 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:37.699901 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-afbe5b-openshift-default-65fbcd9cf5-78knp"] Oct 30 22:05:38.573539 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:38.573506 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f10eae6b-051d-4929-902f-abcd38146a15" path="/var/lib/kubelet/pods/f10eae6b-051d-4929-902f-abcd38146a15/volumes" Oct 30 22:05:47.626241 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:47.626210 4560 scope.go:122] "RemoveContainer" containerID="1e41a2699cba177342cfd9a7ea0aef1b76bd8a4e34c342db72ecaa63cb435404" Oct 30 22:05:47.634600 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:47.634582 4560 scope.go:122] "RemoveContainer" containerID="7f4bf1cabc1afb82734f606818b715f74ec2ff0eba2631cf589a68205ffb4f24" Oct 30 22:05:47.642013 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:47.641993 4560 scope.go:122] "RemoveContainer" containerID="b8b11d7dff2712a080062df716af51f2f06862f133dd4a2cdfd164e24d676325" Oct 30 22:05:47.649315 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:05:47.649297 4560 scope.go:122] "RemoveContainer" containerID="94b2c9d4ed0f0badd36f6220afba88e75d356f0541f8fa5fff13c9c54bf49fe2" Oct 30 22:05:49.683882 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:05:49.683847 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:06:02.233788 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.233746 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6jbjv"] Oct 30 22:06:02.234302 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.234196 4560 cpu_manager.go:414] "containerMap: removing container" podUID="024e6f83-289f-499a-be2d-5aed8a5a6c6d" containerName="payload-processing" Oct 30 22:06:02.234302 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.234214 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="024e6f83-289f-499a-be2d-5aed8a5a6c6d" containerName="payload-processing" Oct 30 22:06:02.234302 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.234229 4560 cpu_manager.go:414] "containerMap: removing container" podUID="90bfb1e4-9526-40f6-aed4-ee46c2d86471" containerName="main" Oct 30 22:06:02.234302 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.234236 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90bfb1e4-9526-40f6-aed4-ee46c2d86471" containerName="main" Oct 30 22:06:02.234302 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.234247 4560 cpu_manager.go:414] "containerMap: removing container" podUID="a59a0fc3-8316-47eb-a42e-552d202b1ab7" containerName="main" Oct 30 22:06:02.234302 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.234254 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a59a0fc3-8316-47eb-a42e-552d202b1ab7" containerName="main" Oct 30 22:06:02.234302 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.234283 4560 cpu_manager.go:414] "containerMap: removing container" podUID="f10eae6b-051d-4929-902f-abcd38146a15" containerName="istio-proxy" Oct 30 22:06:02.234302 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.234290 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f10eae6b-051d-4929-902f-abcd38146a15" containerName="istio-proxy" Oct 30 22:06:02.234769 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.234309 4560 cpu_manager.go:414] "containerMap: removing container" podUID="1807c151-3102-4158-b028-d7dd697f4ad6" containerName="payload-pre-processing" Oct 30 22:06:02.234769 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.234319 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1807c151-3102-4158-b028-d7dd697f4ad6" containerName="payload-pre-processing" Oct 30 22:06:02.234769 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.234330 4560 cpu_manager.go:414] "containerMap: removing container" podUID="971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d" containerName="main" Oct 30 22:06:02.234769 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.234354 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d" containerName="main" Oct 30 22:06:02.234769 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.234367 4560 cpu_manager.go:414] "containerMap: removing container" podUID="9e2d7535-075d-4211-b814-35d6abff7605" containerName="main" Oct 30 22:06:02.234769 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.234374 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e2d7535-075d-4211-b814-35d6abff7605" containerName="main" Oct 30 22:06:02.234769 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.234387 4560 cpu_manager.go:414] "containerMap: removing container" podUID="ef6aad01-3c06-4cb5-9e33-a43d39a5b917" containerName="revoke-keys" Oct 30 22:06:02.234769 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.234394 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef6aad01-3c06-4cb5-9e33-a43d39a5b917" containerName="revoke-keys" Oct 30 22:06:02.234769 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.234481 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="f10eae6b-051d-4929-902f-abcd38146a15" containerName="istio-proxy" Oct 30 22:06:02.234769 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.234496 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="024e6f83-289f-499a-be2d-5aed8a5a6c6d" containerName="payload-processing" Oct 30 22:06:02.234769 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.234507 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="a59a0fc3-8316-47eb-a42e-552d202b1ab7" containerName="main" Oct 30 22:06:02.234769 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.234516 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="90bfb1e4-9526-40f6-aed4-ee46c2d86471" containerName="main" Oct 30 22:06:02.234769 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.234525 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="1807c151-3102-4158-b028-d7dd697f4ad6" containerName="payload-pre-processing" Oct 30 22:06:02.234769 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.234535 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="971ed2b8-e2df-4df0-a240-4c3f0c0d7c0d" containerName="main" Oct 30 22:06:02.234769 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.234545 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e2d7535-075d-4211-b814-35d6abff7605" containerName="main" Oct 30 22:06:02.234769 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.234554 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef6aad01-3c06-4cb5-9e33-a43d39a5b917" containerName="revoke-keys" Oct 30 22:06:02.239685 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.239664 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6jbjv" Oct 30 22:06:02.336691 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.336633 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:06:02.336691 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.336664 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:06:02.336904 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.336759 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:06:02.336904 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.336762 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:06:02.338562 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.338541 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6jbjv"] Oct 30 22:06:02.368777 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.368737 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtm7p\" (UniqueName: \"kubernetes.io/projected/6ae694ad-70a9-4f09-86d6-1b892d088c46-kube-api-access-jtm7p\") pod \"maas-api-revoke-sub-85237e45-6jbjv\" (UID: \"6ae694ad-70a9-4f09-86d6-1b892d088c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6jbjv" Oct 30 22:06:02.368954 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.368781 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ae694ad-70a9-4f09-86d6-1b892d088c46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-6jbjv\" (UID: \"6ae694ad-70a9-4f09-86d6-1b892d088c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6jbjv" Oct 30 22:06:02.368954 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.368899 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ae694ad-70a9-4f09-86d6-1b892d088c46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-6jbjv\" (UID: \"6ae694ad-70a9-4f09-86d6-1b892d088c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6jbjv" Oct 30 22:06:02.469653 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.469611 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ae694ad-70a9-4f09-86d6-1b892d088c46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-6jbjv\" (UID: \"6ae694ad-70a9-4f09-86d6-1b892d088c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6jbjv" Oct 30 22:06:02.469864 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.469708 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtm7p\" (UniqueName: \"kubernetes.io/projected/6ae694ad-70a9-4f09-86d6-1b892d088c46-kube-api-access-jtm7p\") pod \"maas-api-revoke-sub-85237e45-6jbjv\" (UID: \"6ae694ad-70a9-4f09-86d6-1b892d088c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6jbjv" Oct 30 22:06:02.469864 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.469732 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ae694ad-70a9-4f09-86d6-1b892d088c46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-6jbjv\" (UID: \"6ae694ad-70a9-4f09-86d6-1b892d088c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6jbjv" Oct 30 22:06:02.470429 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.470408 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ae694ad-70a9-4f09-86d6-1b892d088c46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-6jbjv\" (UID: \"6ae694ad-70a9-4f09-86d6-1b892d088c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6jbjv" Oct 30 22:06:02.471963 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.471936 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ae694ad-70a9-4f09-86d6-1b892d088c46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-6jbjv\" (UID: \"6ae694ad-70a9-4f09-86d6-1b892d088c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6jbjv" Oct 30 22:06:02.528511 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.528440 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtm7p\" (UniqueName: \"kubernetes.io/projected/6ae694ad-70a9-4f09-86d6-1b892d088c46-kube-api-access-jtm7p\") pod \"maas-api-revoke-sub-85237e45-6jbjv\" (UID: \"6ae694ad-70a9-4f09-86d6-1b892d088c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6jbjv" Oct 30 22:06:02.551317 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.551286 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6jbjv" Oct 30 22:06:02.926284 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:02.926207 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6jbjv"] Oct 30 22:06:02.932042 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:06:02.932013 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae694ad_70a9_4f09_86d6_1b892d088c46.slice/crio-0f9a2d8095d9651969e103495c317b7535523bbda9ca7467926ae40b63f7cfc2 WatchSource:0}: Error finding container 0f9a2d8095d9651969e103495c317b7535523bbda9ca7467926ae40b63f7cfc2: Status 404 returned error can't find the container with id 0f9a2d8095d9651969e103495c317b7535523bbda9ca7467926ae40b63f7cfc2 Oct 30 22:06:03.643969 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:03.643890 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6jbjv" event={"ID":"6ae694ad-70a9-4f09-86d6-1b892d088c46","Type":"ContainerStarted","Data":"2c4f71935e075fa6bf66ab4aaf8beb62162c284b5ca08352acd0aa5a03c27299"} Oct 30 22:06:03.643969 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:03.643944 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6jbjv" event={"ID":"6ae694ad-70a9-4f09-86d6-1b892d088c46","Type":"ContainerStarted","Data":"0f9a2d8095d9651969e103495c317b7535523bbda9ca7467926ae40b63f7cfc2"} Oct 30 22:06:03.755028 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:03.754977 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6jbjv" podStartSLOduration=1.75496488 podStartE2EDuration="1.75496488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:03.753961828 +0000 UTC m=+3139.308661606" watchObservedRunningTime="2026-10-30 22:06:03.75496488 +0000 UTC m=+3139.309664656" Oct 30 22:06:04.651868 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:04.651823 4560 generic.go:356] "Generic (PLEG): container finished" podID="6ae694ad-70a9-4f09-86d6-1b892d088c46" containerID="2c4f71935e075fa6bf66ab4aaf8beb62162c284b5ca08352acd0aa5a03c27299" exitCode=0 Oct 30 22:06:04.652291 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:04.651899 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6jbjv" event={"ID":"6ae694ad-70a9-4f09-86d6-1b892d088c46","Type":"ContainerDied","Data":"2c4f71935e075fa6bf66ab4aaf8beb62162c284b5ca08352acd0aa5a03c27299"} Oct 30 22:06:05.784528 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:05.784502 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6jbjv" Oct 30 22:06:05.899725 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:05.899695 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ae694ad-70a9-4f09-86d6-1b892d088c46-kube-api-access-jtm7p\" (UniqueName: \"kubernetes.io/projected/6ae694ad-70a9-4f09-86d6-1b892d088c46-kube-api-access-jtm7p\") pod \"6ae694ad-70a9-4f09-86d6-1b892d088c46\" (UID: \"6ae694ad-70a9-4f09-86d6-1b892d088c46\") " Oct 30 22:06:05.899906 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:05.899799 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ae694ad-70a9-4f09-86d6-1b892d088c46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ae694ad-70a9-4f09-86d6-1b892d088c46-maas-api-openshift-ca\") pod \"6ae694ad-70a9-4f09-86d6-1b892d088c46\" (UID: \"6ae694ad-70a9-4f09-86d6-1b892d088c46\") " Oct 30 22:06:05.899906 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:05.899826 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ae694ad-70a9-4f09-86d6-1b892d088c46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ae694ad-70a9-4f09-86d6-1b892d088c46-maas-api-xks-ca\") pod \"6ae694ad-70a9-4f09-86d6-1b892d088c46\" (UID: \"6ae694ad-70a9-4f09-86d6-1b892d088c46\") " Oct 30 22:06:05.900207 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:05.900181 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ae694ad-70a9-4f09-86d6-1b892d088c46-maas-api-openshift-ca" pod "6ae694ad-70a9-4f09-86d6-1b892d088c46" (UID: "6ae694ad-70a9-4f09-86d6-1b892d088c46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:06:05.902008 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:05.901980 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ae694ad-70a9-4f09-86d6-1b892d088c46-maas-api-xks-ca" pod "6ae694ad-70a9-4f09-86d6-1b892d088c46" (UID: "6ae694ad-70a9-4f09-86d6-1b892d088c46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:06:05.902266 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:05.902244 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae694ad-70a9-4f09-86d6-1b892d088c46-kube-api-access-jtm7p" pod "6ae694ad-70a9-4f09-86d6-1b892d088c46" (UID: "6ae694ad-70a9-4f09-86d6-1b892d088c46"). InnerVolumeSpecName "kube-api-access-jtm7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:06:06.000856 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:06.000822 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ae694ad-70a9-4f09-86d6-1b892d088c46-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:06.000856 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:06.000850 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ae694ad-70a9-4f09-86d6-1b892d088c46-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:06.000856 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:06.000859 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtm7p\" (UniqueName: \"kubernetes.io/projected/6ae694ad-70a9-4f09-86d6-1b892d088c46-kube-api-access-jtm7p\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:06.662144 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:06.662112 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6jbjv" Oct 30 22:06:06.662319 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:06.662140 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6jbjv" event={"ID":"6ae694ad-70a9-4f09-86d6-1b892d088c46","Type":"ContainerDied","Data":"0f9a2d8095d9651969e103495c317b7535523bbda9ca7467926ae40b63f7cfc2"} Oct 30 22:06:06.662319 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:06.662194 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f9a2d8095d9651969e103495c317b7535523bbda9ca7467926ae40b63f7cfc2" Oct 30 22:06:19.683884 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:06:19.683844 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:06:47.705057 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:47.705024 4560 scope.go:122] "RemoveContainer" containerID="36c6711f316c4ebe76ae889fa17ee363a0954b6d4ce27043f5fb27ae40ea59b1" Oct 30 22:06:47.713587 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:47.713569 4560 scope.go:122] "RemoveContainer" containerID="73de2528aad288f3ff8505acac417133efe07a9cd2829480460b4b7da5e29ff4" Oct 30 22:06:47.721327 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:47.721301 4560 scope.go:122] "RemoveContainer" containerID="a414703a772e27712f410033cbec2271cb1a5235da709c114d6b89c4ddd54e06" Oct 30 22:06:47.728917 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:47.728895 4560 scope.go:122] "RemoveContainer" containerID="a445c0223c04cd4fb73dda7a38014e56f31062b45ea7ecb33faa691ac89870fb" Oct 30 22:06:49.684391 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:06:49.684334 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:06:50.048786 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:50.048744 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-816ca17c-l2249"] Oct 30 22:06:50.096173 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:50.096096 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfe1cd9b-dxgzt"] Oct 30 22:06:50.097857 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:50.097834 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-816ca17c-l2249"] Oct 30 22:06:50.099134 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:50.099114 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfe1cd9b-dxgzt"] Oct 30 22:06:50.574175 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:50.574143 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11aa0088-fac9-4ff4-a925-e241e613137f" path="/var/lib/kubelet/pods/11aa0088-fac9-4ff4-a925-e241e613137f/volumes" Oct 30 22:06:50.574556 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:50.574542 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b0e07f4-337a-4961-aa14-36f1c8989433" path="/var/lib/kubelet/pods/2b0e07f4-337a-4961-aa14-36f1c8989433/volumes" Oct 30 22:06:51.047851 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:51.047819 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c34bf965-stnn7"] Oct 30 22:06:51.050255 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:51.050234 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd387d1e-lpzbb"] Oct 30 22:06:51.094755 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:51.094723 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c34bf965-stnn7"] Oct 30 22:06:51.096121 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:51.096092 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd387d1e-lpzbb"] Oct 30 22:06:52.573804 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:52.573772 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="282c407f-8be5-4c81-9d4c-e67f7ba44cc9" path="/var/lib/kubelet/pods/282c407f-8be5-4c81-9d4c-e67f7ba44cc9/volumes" Oct 30 22:06:52.576301 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:06:52.574157 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ee952bc-3471-4387-8b50-b242f8dc9b44" path="/var/lib/kubelet/pods/5ee952bc-3471-4387-8b50-b242f8dc9b44/volumes" Oct 30 22:07:02.964900 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:02.964813 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:07:11.414947 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:11.414910 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-zx45x"] Oct 30 22:07:11.415472 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:11.415214 4560 cpu_manager.go:414] "containerMap: removing container" podUID="6ae694ad-70a9-4f09-86d6-1b892d088c46" containerName="revoke-keys" Oct 30 22:07:11.415472 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:11.415224 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ae694ad-70a9-4f09-86d6-1b892d088c46" containerName="revoke-keys" Oct 30 22:07:11.415472 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:11.415283 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ae694ad-70a9-4f09-86d6-1b892d088c46" containerName="revoke-keys" Oct 30 22:07:11.418539 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:11.418523 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-zx45x" Oct 30 22:07:11.467666 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:11.467640 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7g8r\" (UniqueName: \"kubernetes.io/projected/7a0e9987-d411-4368-a2e1-1217505619e0-kube-api-access-k7g8r\") pod \"maas-api-revoke-sub-c1e271a9-zx45x\" (UID: \"7a0e9987-d411-4368-a2e1-1217505619e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-zx45x" Oct 30 22:07:11.467803 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:11.467701 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a0e9987-d411-4368-a2e1-1217505619e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-zx45x\" (UID: \"7a0e9987-d411-4368-a2e1-1217505619e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-zx45x" Oct 30 22:07:11.467803 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:11.467772 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:07:11.467803 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:11.467772 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a0e9987-d411-4368-a2e1-1217505619e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-zx45x\" (UID: \"7a0e9987-d411-4368-a2e1-1217505619e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-zx45x" Oct 30 22:07:11.514749 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:11.514721 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:07:11.514749 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:11.514743 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:07:11.514921 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:11.514809 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:07:11.516564 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:11.516545 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-zx45x"] Oct 30 22:07:11.568892 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:11.568866 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a0e9987-d411-4368-a2e1-1217505619e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-zx45x\" (UID: \"7a0e9987-d411-4368-a2e1-1217505619e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-zx45x" Oct 30 22:07:11.568892 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:11.568892 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a0e9987-d411-4368-a2e1-1217505619e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-zx45x\" (UID: \"7a0e9987-d411-4368-a2e1-1217505619e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-zx45x" Oct 30 22:07:11.569064 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:11.568955 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7g8r\" (UniqueName: \"kubernetes.io/projected/7a0e9987-d411-4368-a2e1-1217505619e0-kube-api-access-k7g8r\") pod \"maas-api-revoke-sub-c1e271a9-zx45x\" (UID: \"7a0e9987-d411-4368-a2e1-1217505619e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-zx45x" Oct 30 22:07:11.569517 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:11.569498 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a0e9987-d411-4368-a2e1-1217505619e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-zx45x\" (UID: \"7a0e9987-d411-4368-a2e1-1217505619e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-zx45x" Oct 30 22:07:11.571172 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:11.571154 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a0e9987-d411-4368-a2e1-1217505619e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-zx45x\" (UID: \"7a0e9987-d411-4368-a2e1-1217505619e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-zx45x" Oct 30 22:07:11.627930 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:11.627902 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7g8r\" (UniqueName: \"kubernetes.io/projected/7a0e9987-d411-4368-a2e1-1217505619e0-kube-api-access-k7g8r\") pod \"maas-api-revoke-sub-c1e271a9-zx45x\" (UID: \"7a0e9987-d411-4368-a2e1-1217505619e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-zx45x" Oct 30 22:07:11.730289 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:11.730268 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-zx45x" Oct 30 22:07:12.098956 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:12.098883 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-zx45x"] Oct 30 22:07:12.105247 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:07:12.105221 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a0e9987_d411_4368_a2e1_1217505619e0.slice/crio-f62c7b8937180f04536f889f103d79a443df503becba063dc436c6ddb66bf917 WatchSource:0}: Error finding container f62c7b8937180f04536f889f103d79a443df503becba063dc436c6ddb66bf917: Status 404 returned error can't find the container with id f62c7b8937180f04536f889f103d79a443df503becba063dc436c6ddb66bf917 Oct 30 22:07:13.011634 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:13.011605 4560 generic.go:356] "Generic (PLEG): container finished" podID="7a0e9987-d411-4368-a2e1-1217505619e0" containerID="adf6fb03bafcbb70495587d214b3b42c683c98550b3112a6e3549c7a3b17b3df" exitCode=0 Oct 30 22:07:13.012044 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:13.011657 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-zx45x" event={"ID":"7a0e9987-d411-4368-a2e1-1217505619e0","Type":"ContainerDied","Data":"adf6fb03bafcbb70495587d214b3b42c683c98550b3112a6e3549c7a3b17b3df"} Oct 30 22:07:13.012044 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:13.011720 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-zx45x" event={"ID":"7a0e9987-d411-4368-a2e1-1217505619e0","Type":"ContainerStarted","Data":"f62c7b8937180f04536f889f103d79a443df503becba063dc436c6ddb66bf917"} Oct 30 22:07:14.146973 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:14.146948 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-zx45x" Oct 30 22:07:14.191605 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:14.191573 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a0e9987-d411-4368-a2e1-1217505619e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a0e9987-d411-4368-a2e1-1217505619e0-maas-api-xks-ca\") pod \"7a0e9987-d411-4368-a2e1-1217505619e0\" (UID: \"7a0e9987-d411-4368-a2e1-1217505619e0\") " Oct 30 22:07:14.191803 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:14.191628 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a0e9987-d411-4368-a2e1-1217505619e0-kube-api-access-k7g8r\" (UniqueName: \"kubernetes.io/projected/7a0e9987-d411-4368-a2e1-1217505619e0-kube-api-access-k7g8r\") pod \"7a0e9987-d411-4368-a2e1-1217505619e0\" (UID: \"7a0e9987-d411-4368-a2e1-1217505619e0\") " Oct 30 22:07:14.191803 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:14.191653 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a0e9987-d411-4368-a2e1-1217505619e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a0e9987-d411-4368-a2e1-1217505619e0-maas-api-openshift-ca\") pod \"7a0e9987-d411-4368-a2e1-1217505619e0\" (UID: \"7a0e9987-d411-4368-a2e1-1217505619e0\") " Oct 30 22:07:14.192085 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:14.192056 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a0e9987-d411-4368-a2e1-1217505619e0-maas-api-openshift-ca" pod "7a0e9987-d411-4368-a2e1-1217505619e0" (UID: "7a0e9987-d411-4368-a2e1-1217505619e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:14.193585 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:14.193565 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a0e9987-d411-4368-a2e1-1217505619e0-maas-api-xks-ca" pod "7a0e9987-d411-4368-a2e1-1217505619e0" (UID: "7a0e9987-d411-4368-a2e1-1217505619e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:07:14.193654 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:14.193639 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a0e9987-d411-4368-a2e1-1217505619e0-kube-api-access-k7g8r" pod "7a0e9987-d411-4368-a2e1-1217505619e0" (UID: "7a0e9987-d411-4368-a2e1-1217505619e0"). InnerVolumeSpecName "kube-api-access-k7g8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:14.292573 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:14.292506 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a0e9987-d411-4368-a2e1-1217505619e0-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:14.292573 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:14.292532 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7g8r\" (UniqueName: \"kubernetes.io/projected/7a0e9987-d411-4368-a2e1-1217505619e0-kube-api-access-k7g8r\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:14.292573 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:14.292542 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a0e9987-d411-4368-a2e1-1217505619e0-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:15.023723 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:15.023695 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-zx45x" Oct 30 22:07:15.023883 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:15.023731 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-zx45x" event={"ID":"7a0e9987-d411-4368-a2e1-1217505619e0","Type":"ContainerDied","Data":"f62c7b8937180f04536f889f103d79a443df503becba063dc436c6ddb66bf917"} Oct 30 22:07:15.023883 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:15.023780 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f62c7b8937180f04536f889f103d79a443df503becba063dc436c6ddb66bf917" Oct 30 22:07:18.058823 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:18.058789 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5f4fd8f-hrps7"] Oct 30 22:07:18.105971 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:18.105934 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d0bdf9b-gv5rp"] Oct 30 22:07:18.107286 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:18.107268 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5f4fd8f-hrps7"] Oct 30 22:07:18.108514 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:18.108494 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d0bdf9b-gv5rp"] Oct 30 22:07:18.573753 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:18.573723 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eecc880-6f0d-4a2c-888e-32803ef8c86b" path="/var/lib/kubelet/pods/2eecc880-6f0d-4a2c-888e-32803ef8c86b/volumes" Oct 30 22:07:18.574263 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:18.574241 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81ad1290-d768-489d-9bcf-2e8f9214b369" path="/var/lib/kubelet/pods/81ad1290-d768-489d-9bcf-2e8f9214b369/volumes" Oct 30 22:07:19.683381 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:07:19.683327 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:07:24.352458 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:24.352420 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:07:35.393359 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:35.393305 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:07:40.045817 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:40.045783 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23f42261-z5dx2"] Oct 30 22:07:40.092542 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:40.092509 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23f42261-z5dx2"] Oct 30 22:07:40.573705 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:40.573670 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4b81eaf-72d9-4ed8-8325-52a8a31308f2" path="/var/lib/kubelet/pods/c4b81eaf-72d9-4ed8-8325-52a8a31308f2/volumes" Oct 30 22:07:41.051409 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:41.051372 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e289e3d-ttgz2"] Oct 30 22:07:41.097817 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:41.097788 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e289e3d-ttgz2"] Oct 30 22:07:42.574270 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:42.574233 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d860f97c-a48b-4848-8512-79351fb19f37" path="/var/lib/kubelet/pods/d860f97c-a48b-4848-8512-79351fb19f37/volumes" Oct 30 22:07:42.822520 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:42.822486 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2wtqm"] Oct 30 22:07:42.822789 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:42.822777 4560 cpu_manager.go:414] "containerMap: removing container" podUID="7a0e9987-d411-4368-a2e1-1217505619e0" containerName="revoke-keys" Oct 30 22:07:42.822833 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:42.822790 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a0e9987-d411-4368-a2e1-1217505619e0" containerName="revoke-keys" Oct 30 22:07:42.822869 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:42.822864 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a0e9987-d411-4368-a2e1-1217505619e0" containerName="revoke-keys" Oct 30 22:07:42.827448 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:42.827392 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2wtqm" Oct 30 22:07:42.925721 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:42.925696 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:07:42.925721 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:42.925720 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:07:42.925928 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:42.925749 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:07:42.925928 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:42.925819 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:07:42.927294 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:42.927276 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2wtqm"] Oct 30 22:07:42.975303 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:42.975278 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7fk9\" (UniqueName: \"kubernetes.io/projected/da34e991-e0f9-46b0-9d6b-9e4219b388f8-kube-api-access-s7fk9\") pod \"maas-api-revoke-sub-2e1f5b5f-2wtqm\" (UID: \"da34e991-e0f9-46b0-9d6b-9e4219b388f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2wtqm" Oct 30 22:07:42.975439 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:42.975331 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da34e991-e0f9-46b0-9d6b-9e4219b388f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-2wtqm\" (UID: \"da34e991-e0f9-46b0-9d6b-9e4219b388f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2wtqm" Oct 30 22:07:42.975495 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:42.975433 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da34e991-e0f9-46b0-9d6b-9e4219b388f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-2wtqm\" (UID: \"da34e991-e0f9-46b0-9d6b-9e4219b388f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2wtqm" Oct 30 22:07:43.076427 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:43.076392 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7fk9\" (UniqueName: \"kubernetes.io/projected/da34e991-e0f9-46b0-9d6b-9e4219b388f8-kube-api-access-s7fk9\") pod \"maas-api-revoke-sub-2e1f5b5f-2wtqm\" (UID: \"da34e991-e0f9-46b0-9d6b-9e4219b388f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2wtqm" Oct 30 22:07:43.076602 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:43.076451 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da34e991-e0f9-46b0-9d6b-9e4219b388f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-2wtqm\" (UID: \"da34e991-e0f9-46b0-9d6b-9e4219b388f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2wtqm" Oct 30 22:07:43.076602 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:43.076484 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da34e991-e0f9-46b0-9d6b-9e4219b388f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-2wtqm\" (UID: \"da34e991-e0f9-46b0-9d6b-9e4219b388f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2wtqm" Oct 30 22:07:43.077211 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:43.077191 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da34e991-e0f9-46b0-9d6b-9e4219b388f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-2wtqm\" (UID: \"da34e991-e0f9-46b0-9d6b-9e4219b388f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2wtqm" Oct 30 22:07:43.078953 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:43.078911 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da34e991-e0f9-46b0-9d6b-9e4219b388f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-2wtqm\" (UID: \"da34e991-e0f9-46b0-9d6b-9e4219b388f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2wtqm" Oct 30 22:07:43.134667 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:43.134642 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7fk9\" (UniqueName: \"kubernetes.io/projected/da34e991-e0f9-46b0-9d6b-9e4219b388f8-kube-api-access-s7fk9\") pod \"maas-api-revoke-sub-2e1f5b5f-2wtqm\" (UID: \"da34e991-e0f9-46b0-9d6b-9e4219b388f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2wtqm" Oct 30 22:07:43.139427 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:43.139412 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2wtqm" Oct 30 22:07:43.508152 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:43.508123 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2wtqm"] Oct 30 22:07:43.515554 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:07:43.515529 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda34e991_e0f9_46b0_9d6b_9e4219b388f8.slice/crio-618437233a20e3d9bf3d227328b9bdf5ee6578387059ce2060a2bdbf13ab94bd WatchSource:0}: Error finding container 618437233a20e3d9bf3d227328b9bdf5ee6578387059ce2060a2bdbf13ab94bd: Status 404 returned error can't find the container with id 618437233a20e3d9bf3d227328b9bdf5ee6578387059ce2060a2bdbf13ab94bd Oct 30 22:07:44.170519 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:44.170484 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2wtqm" event={"ID":"da34e991-e0f9-46b0-9d6b-9e4219b388f8","Type":"ContainerStarted","Data":"594117d3bdf2650f1078c3768e87040fe7ccef98e05e369108250e0350325c4d"} Oct 30 22:07:44.171007 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:44.170548 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2wtqm" event={"ID":"da34e991-e0f9-46b0-9d6b-9e4219b388f8","Type":"ContainerStarted","Data":"618437233a20e3d9bf3d227328b9bdf5ee6578387059ce2060a2bdbf13ab94bd"} Oct 30 22:07:44.328587 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:44.328512 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2wtqm" podStartSLOduration=2.3284986500000002 podStartE2EDuration="2.32849865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:44.280968866 +0000 UTC m=+3239.835668642" watchObservedRunningTime="2026-10-30 22:07:44.32849865 +0000 UTC m=+3239.883198427" Oct 30 22:07:45.177138 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:45.177104 4560 generic.go:356] "Generic (PLEG): container finished" podID="da34e991-e0f9-46b0-9d6b-9e4219b388f8" containerID="594117d3bdf2650f1078c3768e87040fe7ccef98e05e369108250e0350325c4d" exitCode=0 Oct 30 22:07:45.177559 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:45.177190 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2wtqm" event={"ID":"da34e991-e0f9-46b0-9d6b-9e4219b388f8","Type":"ContainerDied","Data":"594117d3bdf2650f1078c3768e87040fe7ccef98e05e369108250e0350325c4d"} Oct 30 22:07:46.316683 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:46.316657 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2wtqm" Oct 30 22:07:46.401272 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:46.401243 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da34e991-e0f9-46b0-9d6b-9e4219b388f8-kube-api-access-s7fk9\" (UniqueName: \"kubernetes.io/projected/da34e991-e0f9-46b0-9d6b-9e4219b388f8-kube-api-access-s7fk9\") pod \"da34e991-e0f9-46b0-9d6b-9e4219b388f8\" (UID: \"da34e991-e0f9-46b0-9d6b-9e4219b388f8\") " Oct 30 22:07:46.401454 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:46.401376 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da34e991-e0f9-46b0-9d6b-9e4219b388f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da34e991-e0f9-46b0-9d6b-9e4219b388f8-maas-api-xks-ca\") pod \"da34e991-e0f9-46b0-9d6b-9e4219b388f8\" (UID: \"da34e991-e0f9-46b0-9d6b-9e4219b388f8\") " Oct 30 22:07:46.401454 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:46.401424 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da34e991-e0f9-46b0-9d6b-9e4219b388f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da34e991-e0f9-46b0-9d6b-9e4219b388f8-maas-api-openshift-ca\") pod \"da34e991-e0f9-46b0-9d6b-9e4219b388f8\" (UID: \"da34e991-e0f9-46b0-9d6b-9e4219b388f8\") " Oct 30 22:07:46.401776 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:46.401753 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da34e991-e0f9-46b0-9d6b-9e4219b388f8-maas-api-openshift-ca" pod "da34e991-e0f9-46b0-9d6b-9e4219b388f8" (UID: "da34e991-e0f9-46b0-9d6b-9e4219b388f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:46.403322 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:46.403300 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da34e991-e0f9-46b0-9d6b-9e4219b388f8-maas-api-xks-ca" pod "da34e991-e0f9-46b0-9d6b-9e4219b388f8" (UID: "da34e991-e0f9-46b0-9d6b-9e4219b388f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:07:46.403322 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:46.403309 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da34e991-e0f9-46b0-9d6b-9e4219b388f8-kube-api-access-s7fk9" pod "da34e991-e0f9-46b0-9d6b-9e4219b388f8" (UID: "da34e991-e0f9-46b0-9d6b-9e4219b388f8"). InnerVolumeSpecName "kube-api-access-s7fk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:46.502346 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:46.502318 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7fk9\" (UniqueName: \"kubernetes.io/projected/da34e991-e0f9-46b0-9d6b-9e4219b388f8-kube-api-access-s7fk9\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:46.502491 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:46.502364 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da34e991-e0f9-46b0-9d6b-9e4219b388f8-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:46.502491 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:46.502374 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da34e991-e0f9-46b0-9d6b-9e4219b388f8-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:47.188262 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:47.188229 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2wtqm" Oct 30 22:07:47.188463 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:47.188221 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2wtqm" event={"ID":"da34e991-e0f9-46b0-9d6b-9e4219b388f8","Type":"ContainerDied","Data":"618437233a20e3d9bf3d227328b9bdf5ee6578387059ce2060a2bdbf13ab94bd"} Oct 30 22:07:47.188463 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:47.188387 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="618437233a20e3d9bf3d227328b9bdf5ee6578387059ce2060a2bdbf13ab94bd" Oct 30 22:07:47.757728 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:47.757696 4560 scope.go:122] "RemoveContainer" containerID="5be1a42105ef388be038cc01d603f0b7532978e5def79a215a2fdcc45776e0fe" Oct 30 22:07:47.766228 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:47.766201 4560 scope.go:122] "RemoveContainer" containerID="c79dde18f6f83bfc9a1925c70476c80b921c98e8348f598494bebde3746f59d9" Oct 30 22:07:47.773715 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:47.773700 4560 scope.go:122] "RemoveContainer" containerID="5bc41345b066f183450dfafd3942c3c4d8cf8845bfa7d7c71218ce71b0056d34" Oct 30 22:07:47.781013 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:47.780995 4560 scope.go:122] "RemoveContainer" containerID="f1509cae3d313d33ceb8a50c0c8dcda43c31a3f20c4302854ab7c6372dc1db1d" Oct 30 22:07:47.788059 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:47.788044 4560 scope.go:122] "RemoveContainer" containerID="fc3933595d4530175cfb00191bdb08b35869c477a28af454d450895e2ccdbc85" Oct 30 22:07:47.795055 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:47.795032 4560 scope.go:122] "RemoveContainer" containerID="5b19aec620a7bc0f4ad91c30083fa4d7347e826c288720df4771bea3bc68c32f" Oct 30 22:07:47.801760 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:47.801738 4560 scope.go:122] "RemoveContainer" containerID="8a2e70151d8c4cb0236662dfbc6b5a628eeeb7c4f9efe9bc5b749ac962a023c4" Oct 30 22:07:49.682736 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:07:49.682702 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:07:55.347402 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:55.347364 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mmk6d"] Oct 30 22:07:55.347792 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:55.347668 4560 cpu_manager.go:414] "containerMap: removing container" podUID="da34e991-e0f9-46b0-9d6b-9e4219b388f8" containerName="revoke-keys" Oct 30 22:07:55.347792 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:55.347678 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da34e991-e0f9-46b0-9d6b-9e4219b388f8" containerName="revoke-keys" Oct 30 22:07:55.347792 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:55.347751 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="da34e991-e0f9-46b0-9d6b-9e4219b388f8" containerName="revoke-keys" Oct 30 22:07:55.352795 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:55.352771 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mmk6d" Oct 30 22:07:55.364489 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:55.364466 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-566cl\" (UniqueName: \"kubernetes.io/projected/b02703c7-3fd8-4292-a707-62f1911ae813-kube-api-access-566cl\") pod \"maas-api-revoke-sub-ec843018-mmk6d\" (UID: \"b02703c7-3fd8-4292-a707-62f1911ae813\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mmk6d" Oct 30 22:07:55.364615 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:55.364532 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b02703c7-3fd8-4292-a707-62f1911ae813-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-mmk6d\" (UID: \"b02703c7-3fd8-4292-a707-62f1911ae813\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mmk6d" Oct 30 22:07:55.364691 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:55.364639 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b02703c7-3fd8-4292-a707-62f1911ae813-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-mmk6d\" (UID: \"b02703c7-3fd8-4292-a707-62f1911ae813\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mmk6d" Oct 30 22:07:55.406168 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:55.406137 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mmk6d"] Oct 30 22:07:55.450909 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:55.450881 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:07:55.450909 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:55.450881 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:07:55.451083 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:55.450912 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:07:55.451083 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:55.450928 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:07:55.465971 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:55.465945 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b02703c7-3fd8-4292-a707-62f1911ae813-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-mmk6d\" (UID: \"b02703c7-3fd8-4292-a707-62f1911ae813\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mmk6d" Oct 30 22:07:55.466088 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:55.465985 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b02703c7-3fd8-4292-a707-62f1911ae813-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-mmk6d\" (UID: \"b02703c7-3fd8-4292-a707-62f1911ae813\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mmk6d" Oct 30 22:07:55.466088 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:55.466024 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-566cl\" (UniqueName: \"kubernetes.io/projected/b02703c7-3fd8-4292-a707-62f1911ae813-kube-api-access-566cl\") pod \"maas-api-revoke-sub-ec843018-mmk6d\" (UID: \"b02703c7-3fd8-4292-a707-62f1911ae813\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mmk6d" Oct 30 22:07:55.466616 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:55.466600 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b02703c7-3fd8-4292-a707-62f1911ae813-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-mmk6d\" (UID: \"b02703c7-3fd8-4292-a707-62f1911ae813\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mmk6d" Oct 30 22:07:55.468321 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:55.468306 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b02703c7-3fd8-4292-a707-62f1911ae813-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-mmk6d\" (UID: \"b02703c7-3fd8-4292-a707-62f1911ae813\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mmk6d" Oct 30 22:07:55.524863 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:55.524836 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-566cl\" (UniqueName: \"kubernetes.io/projected/b02703c7-3fd8-4292-a707-62f1911ae813-kube-api-access-566cl\") pod \"maas-api-revoke-sub-ec843018-mmk6d\" (UID: \"b02703c7-3fd8-4292-a707-62f1911ae813\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mmk6d" Oct 30 22:07:55.665399 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:55.665308 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mmk6d" Oct 30 22:07:55.702565 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:55.702534 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:07:56.035583 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:56.035549 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mmk6d"] Oct 30 22:07:56.041252 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:07:56.041222 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb02703c7_3fd8_4292_a707_62f1911ae813.slice/crio-1d4ff99613aeb089fec52ee2ee388774dc811566c5691ddecaa885dbe56646a3 WatchSource:0}: Error finding container 1d4ff99613aeb089fec52ee2ee388774dc811566c5691ddecaa885dbe56646a3: Status 404 returned error can't find the container with id 1d4ff99613aeb089fec52ee2ee388774dc811566c5691ddecaa885dbe56646a3 Oct 30 22:07:56.234797 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:56.234766 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mmk6d" event={"ID":"b02703c7-3fd8-4292-a707-62f1911ae813","Type":"ContainerStarted","Data":"1d4ff99613aeb089fec52ee2ee388774dc811566c5691ddecaa885dbe56646a3"} Oct 30 22:07:57.240694 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:57.240659 4560 generic.go:356] "Generic (PLEG): container finished" podID="b02703c7-3fd8-4292-a707-62f1911ae813" containerID="ed902791d6cfacc71f21d7c09089cfc85a592bc7d2a2b6de8ac71e8d75fe04fe" exitCode=0 Oct 30 22:07:57.241099 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:57.240703 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mmk6d" event={"ID":"b02703c7-3fd8-4292-a707-62f1911ae813","Type":"ContainerDied","Data":"ed902791d6cfacc71f21d7c09089cfc85a592bc7d2a2b6de8ac71e8d75fe04fe"} Oct 30 22:07:58.367566 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:58.367544 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mmk6d" Oct 30 22:07:58.391751 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:58.391718 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b02703c7-3fd8-4292-a707-62f1911ae813-kube-api-access-566cl\" (UniqueName: \"kubernetes.io/projected/b02703c7-3fd8-4292-a707-62f1911ae813-kube-api-access-566cl\") pod \"b02703c7-3fd8-4292-a707-62f1911ae813\" (UID: \"b02703c7-3fd8-4292-a707-62f1911ae813\") " Oct 30 22:07:58.391938 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:58.391777 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b02703c7-3fd8-4292-a707-62f1911ae813-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b02703c7-3fd8-4292-a707-62f1911ae813-maas-api-xks-ca\") pod \"b02703c7-3fd8-4292-a707-62f1911ae813\" (UID: \"b02703c7-3fd8-4292-a707-62f1911ae813\") " Oct 30 22:07:58.391938 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:58.391809 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b02703c7-3fd8-4292-a707-62f1911ae813-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b02703c7-3fd8-4292-a707-62f1911ae813-maas-api-openshift-ca\") pod \"b02703c7-3fd8-4292-a707-62f1911ae813\" (UID: \"b02703c7-3fd8-4292-a707-62f1911ae813\") " Oct 30 22:07:58.392238 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:58.392207 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b02703c7-3fd8-4292-a707-62f1911ae813-maas-api-openshift-ca" pod "b02703c7-3fd8-4292-a707-62f1911ae813" (UID: "b02703c7-3fd8-4292-a707-62f1911ae813"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:58.393894 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:58.393865 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b02703c7-3fd8-4292-a707-62f1911ae813-maas-api-xks-ca" pod "b02703c7-3fd8-4292-a707-62f1911ae813" (UID: "b02703c7-3fd8-4292-a707-62f1911ae813"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:07:58.393983 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:58.393925 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b02703c7-3fd8-4292-a707-62f1911ae813-kube-api-access-566cl" pod "b02703c7-3fd8-4292-a707-62f1911ae813" (UID: "b02703c7-3fd8-4292-a707-62f1911ae813"). InnerVolumeSpecName "kube-api-access-566cl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:58.493168 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:58.493135 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-566cl\" (UniqueName: \"kubernetes.io/projected/b02703c7-3fd8-4292-a707-62f1911ae813-kube-api-access-566cl\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:58.493168 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:58.493165 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b02703c7-3fd8-4292-a707-62f1911ae813-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:58.493361 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:58.493178 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b02703c7-3fd8-4292-a707-62f1911ae813-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:59.253721 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:59.253689 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mmk6d" Oct 30 22:07:59.253912 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:59.253690 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mmk6d" event={"ID":"b02703c7-3fd8-4292-a707-62f1911ae813","Type":"ContainerDied","Data":"1d4ff99613aeb089fec52ee2ee388774dc811566c5691ddecaa885dbe56646a3"} Oct 30 22:07:59.253912 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:07:59.253830 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d4ff99613aeb089fec52ee2ee388774dc811566c5691ddecaa885dbe56646a3" Oct 30 22:08:14.492405 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:08:14.492374 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:08:17.181923 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:08:17.181879 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-gl8p4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:08:17.182565 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:08:17.182531 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" podUID="4c21f641-8dd8-4549-8748-94c9fdbd8ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:17.364472 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:08:17.364436 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-m2sxq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:08:17.364767 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:08:17.364735 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" podUID="c73ef128-5710-4654-a155-4b9083ab04bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:19.683402 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:08:19.683367 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:08:47.165753 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:08:47.165719 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:08:47.840427 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:08:47.840390 4560 scope.go:122] "RemoveContainer" containerID="504a74affb73311e04fa9e0ecf28fb73a847493022bb895795490add813769a7" Oct 30 22:08:47.850655 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:08:47.850624 4560 scope.go:122] "RemoveContainer" containerID="3b252fd81d0867b3d3cb86e692a3b1ea3e7e4615a7d1bd5a4178b443d8411ebe" Oct 30 22:08:47.860358 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:08:47.860306 4560 scope.go:122] "RemoveContainer" containerID="5fd1675b885c90c47450267748e6b5d0547212550fba987c98b6841b4b8c68d4" Oct 30 22:08:47.869931 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:08:47.869913 4560 scope.go:122] "RemoveContainer" containerID="4d9ff79d527eeabe41a7c6973ed147bec075871a39e6a0e61c0187fdac6949b9" Oct 30 22:08:47.878013 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:08:47.877980 4560 scope.go:122] "RemoveContainer" containerID="e7e28f7aba6690732cd464b8fd289fd94f65364a45f37e7d5618c5b17a338916" Oct 30 22:08:47.886840 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:08:47.886824 4560 scope.go:122] "RemoveContainer" containerID="f41d81830e6ac88bd13279fd47b762e4a0c3f0393689e6f22d7dbc403adf4671" Oct 30 22:08:49.718623 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:08:49.718588 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:08:54.057240 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:08:54.057208 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ac3eeab-lp8mg"] Oct 30 22:08:54.104008 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:08:54.103978 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ac3eeab-lp8mg"] Oct 30 22:08:54.579274 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:08:54.579243 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81d18e45-cbee-4f35-97df-8f80472ba89a" path="/var/lib/kubelet/pods/81d18e45-cbee-4f35-97df-8f80472ba89a/volumes" Oct 30 22:08:58.090614 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:08:58.090580 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:09:09.108368 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:09:09.107897 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:09:19.684051 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:09:19.684015 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:09:32.806392 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:09:32.800859 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:09:35.048256 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:09:35.048221 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eba46b8-j2fq2"] Oct 30 22:09:35.095488 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:09:35.095445 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7aca459f-5mh87"] Oct 30 22:09:35.096830 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:09:35.096801 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7aca459f-5mh87"] Oct 30 22:09:35.097946 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:09:35.097920 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eba46b8-j2fq2"] Oct 30 22:09:36.574602 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:09:36.574570 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="101812ce-bdc8-484c-ba45-a6c465341d0d" path="/var/lib/kubelet/pods/101812ce-bdc8-484c-ba45-a6c465341d0d/volumes" Oct 30 22:09:36.574996 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:09:36.574955 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82630713-ed8a-4ff9-b7f2-d85385bf69ad" path="/var/lib/kubelet/pods/82630713-ed8a-4ff9-b7f2-d85385bf69ad/volumes" Oct 30 22:09:42.597863 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:09:42.597831 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:09:47.931015 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:09:47.930985 4560 scope.go:122] "RemoveContainer" containerID="99b2ab641d1ccb702e1f1b407d4dc623da12089d8ba78417441f0249c878cdb1" Oct 30 22:09:49.683422 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:09:49.683381 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:10:00.998447 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:10:00.998363 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:10:09.061109 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:10:09.061069 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35b8a54f-zq6zv"] Oct 30 22:10:09.098942 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:10:09.098907 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35b8a54f-zq6zv"] Oct 30 22:10:10.611008 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:10:10.610969 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef6aad01-3c06-4cb5-9e33-a43d39a5b917" path="/var/lib/kubelet/pods/ef6aad01-3c06-4cb5-9e33-a43d39a5b917/volumes" Oct 30 22:10:13.698429 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:10:13.698399 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:10:19.683631 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:10:19.683590 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:10:33.583664 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:10:33.583628 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:10:47.950254 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:10:47.950222 4560 scope.go:122] "RemoveContainer" containerID="7c063e470767e84d7ebc742ba246e9995c44197d99fb5ce3f4fc14b2ab73d8a0" Oct 30 22:10:47.958173 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:10:47.958152 4560 scope.go:122] "RemoveContainer" containerID="a5b6c3fc2437bd23868a7706337e3f4d157ce37d7a95ea273d3e33efe9a80da2" Oct 30 22:10:47.967226 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:10:47.967206 4560 scope.go:122] "RemoveContainer" containerID="0b0591f02672f6c4924d3518a1baf880fc85281f53c3fc2778b6d60157756e6b" Oct 30 22:10:47.974167 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:10:47.974147 4560 scope.go:122] "RemoveContainer" containerID="ca997a0b05e4a7beb50e14fd8fb1dbb7e999f0ec6445261ca9aaf06fc6615c1e" Oct 30 22:10:47.980829 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:10:47.980806 4560 scope.go:122] "RemoveContainer" containerID="16ef64a6710faa33523f9bff454e6bc16069579027fcf1a9b1c8988a7b729300" Oct 30 22:10:49.683942 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:10:49.683902 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:10:55.591521 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:10:55.591482 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:11:06.048449 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:11:06.048413 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6jbjv"] Oct 30 22:11:06.094966 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:11:06.094938 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6jbjv"] Oct 30 22:11:06.574083 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:11:06.574041 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ae694ad-70a9-4f09-86d6-1b892d088c46" path="/var/lib/kubelet/pods/6ae694ad-70a9-4f09-86d6-1b892d088c46/volumes" Oct 30 22:11:08.801096 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:11:08.801056 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:11:11.746511 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:11:11.746470 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-zfpbl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:11:11.746948 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:11:11.746537 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zfpbl" podUID="2e6e5570-8209-4ff2-84d2-1bf88ca77c81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:17.178484 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:11:17.178439 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-gl8p4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:11:17.179108 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:11:17.179074 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gl8p4" podUID="4c21f641-8dd8-4549-8748-94c9fdbd8ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:17.370498 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:11:17.370433 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-m2sxq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:11:17.370851 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:11:17.370817 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" podUID="c73ef128-5710-4654-a155-4b9083ab04bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:19.683514 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:11:19.683473 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:11:19.871639 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:11:19.871605 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:11:42.884352 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:11:42.884307 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:11:48.009231 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:11:48.009190 4560 scope.go:122] "RemoveContainer" containerID="9ddaeddcd172a8d372b9e241ead8f11e1e420769b994cae8b110a34c336f4318" Oct 30 22:11:48.017426 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:11:48.017392 4560 scope.go:122] "RemoveContainer" containerID="00b6a84016c9bdcd1aa3dc1ca2cf2f65f9a5aef36a25e7b28b67eb3eb9009163" Oct 30 22:11:49.684058 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:11:49.684020 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:11:53.361111 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:11:53.361079 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:12:09.888391 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:12:09.888357 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:12:15.046029 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:12:15.045992 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-zx45x"] Oct 30 22:12:15.092924 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:12:15.092892 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-zx45x"] Oct 30 22:12:16.573790 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:12:16.573757 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a0e9987-d411-4368-a2e1-1217505619e0" path="/var/lib/kubelet/pods/7a0e9987-d411-4368-a2e1-1217505619e0/volumes" Oct 30 22:12:19.683705 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:12:19.683667 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:12:31.797814 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:12:31.797772 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:12:42.487480 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:12:42.487441 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:12:47.048580 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:12:47.048544 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2wtqm"] Oct 30 22:12:47.095257 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:12:47.095227 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2wtqm"] Oct 30 22:12:48.036996 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:12:48.036966 4560 scope.go:122] "RemoveContainer" containerID="2c4f71935e075fa6bf66ab4aaf8beb62162c284b5ca08352acd0aa5a03c27299" Oct 30 22:12:48.573375 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:12:48.573320 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da34e991-e0f9-46b0-9d6b-9e4219b388f8" path="/var/lib/kubelet/pods/da34e991-e0f9-46b0-9d6b-9e4219b388f8/volumes" Oct 30 22:12:49.683816 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:12:49.683775 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:12:59.047796 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:12:59.047716 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mmk6d"] Oct 30 22:12:59.094459 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:12:59.094430 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mmk6d"] Oct 30 22:13:00.573536 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:13:00.573500 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b02703c7-3fd8-4292-a707-62f1911ae813" path="/var/lib/kubelet/pods/b02703c7-3fd8-4292-a707-62f1911ae813/volumes" Oct 30 22:13:01.877707 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:13:01.877671 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:13:12.788624 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:13:12.788587 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:13:19.687312 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:13:19.687267 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:13:38.891880 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:13:38.891844 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:13:48.055118 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:13:48.055064 4560 scope.go:122] "RemoveContainer" containerID="594117d3bdf2650f1078c3768e87040fe7ccef98e05e369108250e0350325c4d" Oct 30 22:13:48.068636 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:13:48.068612 4560 scope.go:122] "RemoveContainer" containerID="adf6fb03bafcbb70495587d214b3b42c683c98550b3112a6e3549c7a3b17b3df" Oct 30 22:13:49.682917 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:13:49.682881 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:13:58.090061 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:13:58.090026 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:14:19.684198 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:14:19.684161 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:14:23.507129 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:23.507090 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:14:37.833842 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:37.833765 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14e189aa-mgdk7"] Oct 30 22:14:37.834224 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:37.834057 4560 cpu_manager.go:414] "containerMap: removing container" podUID="b02703c7-3fd8-4292-a707-62f1911ae813" containerName="revoke-keys" Oct 30 22:14:37.834224 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:37.834067 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b02703c7-3fd8-4292-a707-62f1911ae813" containerName="revoke-keys" Oct 30 22:14:37.834224 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:37.834144 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="b02703c7-3fd8-4292-a707-62f1911ae813" containerName="revoke-keys" Oct 30 22:14:37.837086 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:37.837070 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e189aa-mgdk7" Oct 30 22:14:37.886529 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:37.886483 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:14:37.923102 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:37.923067 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6pml\" (UniqueName: \"kubernetes.io/projected/0e212d07-7722-4ba5-ae46-e4f72946c0f0-kube-api-access-f6pml\") pod \"maas-api-revoke-sub-14e189aa-mgdk7\" (UID: \"0e212d07-7722-4ba5-ae46-e4f72946c0f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e189aa-mgdk7" Oct 30 22:14:37.923284 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:37.923120 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e212d07-7722-4ba5-ae46-e4f72946c0f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14e189aa-mgdk7\" (UID: \"0e212d07-7722-4ba5-ae46-e4f72946c0f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e189aa-mgdk7" Oct 30 22:14:37.923284 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:37.923263 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e212d07-7722-4ba5-ae46-e4f72946c0f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14e189aa-mgdk7\" (UID: \"0e212d07-7722-4ba5-ae46-e4f72946c0f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e189aa-mgdk7" Oct 30 22:14:37.933465 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:37.933438 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:14:37.933465 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:37.933447 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:14:37.933649 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:37.933537 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:14:37.981606 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:37.981570 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14e189aa-mgdk7"] Oct 30 22:14:38.024256 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:38.024224 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6pml\" (UniqueName: \"kubernetes.io/projected/0e212d07-7722-4ba5-ae46-e4f72946c0f0-kube-api-access-f6pml\") pod \"maas-api-revoke-sub-14e189aa-mgdk7\" (UID: \"0e212d07-7722-4ba5-ae46-e4f72946c0f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e189aa-mgdk7" Oct 30 22:14:38.024475 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:38.024283 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e212d07-7722-4ba5-ae46-e4f72946c0f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14e189aa-mgdk7\" (UID: \"0e212d07-7722-4ba5-ae46-e4f72946c0f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e189aa-mgdk7" Oct 30 22:14:38.024475 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:38.024326 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e212d07-7722-4ba5-ae46-e4f72946c0f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14e189aa-mgdk7\" (UID: \"0e212d07-7722-4ba5-ae46-e4f72946c0f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e189aa-mgdk7" Oct 30 22:14:38.024981 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:38.024960 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e212d07-7722-4ba5-ae46-e4f72946c0f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14e189aa-mgdk7\" (UID: \"0e212d07-7722-4ba5-ae46-e4f72946c0f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e189aa-mgdk7" Oct 30 22:14:38.026594 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:38.026572 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e212d07-7722-4ba5-ae46-e4f72946c0f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14e189aa-mgdk7\" (UID: \"0e212d07-7722-4ba5-ae46-e4f72946c0f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e189aa-mgdk7" Oct 30 22:14:38.087924 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:38.087851 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6pml\" (UniqueName: \"kubernetes.io/projected/0e212d07-7722-4ba5-ae46-e4f72946c0f0-kube-api-access-f6pml\") pod \"maas-api-revoke-sub-14e189aa-mgdk7\" (UID: \"0e212d07-7722-4ba5-ae46-e4f72946c0f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e189aa-mgdk7" Oct 30 22:14:38.149523 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:38.149481 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e189aa-mgdk7" Oct 30 22:14:38.533918 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:38.533883 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14e189aa-mgdk7"] Oct 30 22:14:38.543280 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:14:38.543237 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e212d07_7722_4ba5_ae46_e4f72946c0f0.slice/crio-dcf9b25a56783dd1b3b56241e6349184e281165d6712d5f5f40a1fd467b8be8b WatchSource:0}: Error finding container dcf9b25a56783dd1b3b56241e6349184e281165d6712d5f5f40a1fd467b8be8b: Status 404 returned error can't find the container with id dcf9b25a56783dd1b3b56241e6349184e281165d6712d5f5f40a1fd467b8be8b Oct 30 22:14:39.040672 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:39.040640 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e189aa-mgdk7" event={"ID":"0e212d07-7722-4ba5-ae46-e4f72946c0f0","Type":"ContainerStarted","Data":"9098dc42dac17882eb646998401c3b0356794d0363e495b30cb34f37d1eed4cb"} Oct 30 22:14:39.041043 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:39.040691 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e189aa-mgdk7" event={"ID":"0e212d07-7722-4ba5-ae46-e4f72946c0f0","Type":"ContainerStarted","Data":"dcf9b25a56783dd1b3b56241e6349184e281165d6712d5f5f40a1fd467b8be8b"} Oct 30 22:14:39.198290 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:39.198239 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e189aa-mgdk7" podStartSLOduration=2.198225671 podStartE2EDuration="2.198225671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:39.197238908 +0000 UTC m=+3654.751938725" watchObservedRunningTime="2026-10-30 22:14:39.198225671 +0000 UTC m=+3654.752925448" Oct 30 22:14:40.045416 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:40.045381 4560 generic.go:356] "Generic (PLEG): container finished" podID="0e212d07-7722-4ba5-ae46-e4f72946c0f0" containerID="9098dc42dac17882eb646998401c3b0356794d0363e495b30cb34f37d1eed4cb" exitCode=0 Oct 30 22:14:40.045854 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:40.045439 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e189aa-mgdk7" event={"ID":"0e212d07-7722-4ba5-ae46-e4f72946c0f0","Type":"ContainerDied","Data":"9098dc42dac17882eb646998401c3b0356794d0363e495b30cb34f37d1eed4cb"} Oct 30 22:14:41.175965 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:41.175935 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e189aa-mgdk7" Oct 30 22:14:41.249808 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:41.249773 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e212d07-7722-4ba5-ae46-e4f72946c0f0-kube-api-access-f6pml\" (UniqueName: \"kubernetes.io/projected/0e212d07-7722-4ba5-ae46-e4f72946c0f0-kube-api-access-f6pml\") pod \"0e212d07-7722-4ba5-ae46-e4f72946c0f0\" (UID: \"0e212d07-7722-4ba5-ae46-e4f72946c0f0\") " Oct 30 22:14:41.250026 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:41.249822 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e212d07-7722-4ba5-ae46-e4f72946c0f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e212d07-7722-4ba5-ae46-e4f72946c0f0-maas-api-openshift-ca\") pod \"0e212d07-7722-4ba5-ae46-e4f72946c0f0\" (UID: \"0e212d07-7722-4ba5-ae46-e4f72946c0f0\") " Oct 30 22:14:41.250026 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:41.249895 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e212d07-7722-4ba5-ae46-e4f72946c0f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e212d07-7722-4ba5-ae46-e4f72946c0f0-maas-api-xks-ca\") pod \"0e212d07-7722-4ba5-ae46-e4f72946c0f0\" (UID: \"0e212d07-7722-4ba5-ae46-e4f72946c0f0\") " Oct 30 22:14:41.250261 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:41.250237 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e212d07-7722-4ba5-ae46-e4f72946c0f0-maas-api-openshift-ca" pod "0e212d07-7722-4ba5-ae46-e4f72946c0f0" (UID: "0e212d07-7722-4ba5-ae46-e4f72946c0f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:41.252021 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:41.251990 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e212d07-7722-4ba5-ae46-e4f72946c0f0-kube-api-access-f6pml" pod "0e212d07-7722-4ba5-ae46-e4f72946c0f0" (UID: "0e212d07-7722-4ba5-ae46-e4f72946c0f0"). InnerVolumeSpecName "kube-api-access-f6pml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:41.252021 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:41.252013 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e212d07-7722-4ba5-ae46-e4f72946c0f0-maas-api-xks-ca" pod "0e212d07-7722-4ba5-ae46-e4f72946c0f0" (UID: "0e212d07-7722-4ba5-ae46-e4f72946c0f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:41.351587 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:41.351490 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e212d07-7722-4ba5-ae46-e4f72946c0f0-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:41.351587 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:41.351528 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6pml\" (UniqueName: \"kubernetes.io/projected/0e212d07-7722-4ba5-ae46-e4f72946c0f0-kube-api-access-f6pml\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:41.351587 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:41.351539 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e212d07-7722-4ba5-ae46-e4f72946c0f0-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:42.054352 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:42.054303 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e189aa-mgdk7" Oct 30 22:14:42.054352 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:42.054327 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e189aa-mgdk7" event={"ID":"0e212d07-7722-4ba5-ae46-e4f72946c0f0","Type":"ContainerDied","Data":"dcf9b25a56783dd1b3b56241e6349184e281165d6712d5f5f40a1fd467b8be8b"} Oct 30 22:14:42.054655 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:42.054392 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcf9b25a56783dd1b3b56241e6349184e281165d6712d5f5f40a1fd467b8be8b" Oct 30 22:14:48.087899 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:48.087874 4560 scope.go:122] "RemoveContainer" containerID="ed902791d6cfacc71f21d7c09089cfc85a592bc7d2a2b6de8ac71e8d75fe04fe" Oct 30 22:14:49.683807 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:14:49.683770 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:14:50.166049 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:50.166014 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-788e42-22pnn"] Oct 30 22:14:50.166324 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:50.166312 4560 cpu_manager.go:414] "containerMap: removing container" podUID="0e212d07-7722-4ba5-ae46-e4f72946c0f0" containerName="revoke-keys" Oct 30 22:14:50.166397 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:50.166325 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e212d07-7722-4ba5-ae46-e4f72946c0f0" containerName="revoke-keys" Oct 30 22:14:50.166435 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:50.166425 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e212d07-7722-4ba5-ae46-e4f72946c0f0" containerName="revoke-keys" Oct 30 22:14:50.169110 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:50.169092 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-788e42-22pnn" Oct 30 22:14:50.222701 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:50.222677 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:14:50.266273 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:50.266246 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:14:50.266592 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:50.266572 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:14:50.267637 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:50.267619 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-788e42-22pnn"] Oct 30 22:14:50.271636 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:50.271622 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:14:50.332388 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:50.332364 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjzdw\" (UniqueName: \"kubernetes.io/projected/29f6a42e-027c-4fb9-9fcd-34b26892d29a-kube-api-access-pjzdw\") pod \"maas-api-revoke-keys-e2e-worker-single-main-788e42-22pnn\" (UID: \"29f6a42e-027c-4fb9-9fcd-34b26892d29a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-788e42-22pnn" Oct 30 22:14:50.332498 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:50.332411 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29f6a42e-027c-4fb9-9fcd-34b26892d29a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-788e42-22pnn\" (UID: \"29f6a42e-027c-4fb9-9fcd-34b26892d29a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-788e42-22pnn" Oct 30 22:14:50.332545 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:50.332505 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29f6a42e-027c-4fb9-9fcd-34b26892d29a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-788e42-22pnn\" (UID: \"29f6a42e-027c-4fb9-9fcd-34b26892d29a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-788e42-22pnn" Oct 30 22:14:50.433690 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:50.433635 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29f6a42e-027c-4fb9-9fcd-34b26892d29a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-788e42-22pnn\" (UID: \"29f6a42e-027c-4fb9-9fcd-34b26892d29a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-788e42-22pnn" Oct 30 22:14:50.433690 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:50.433684 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29f6a42e-027c-4fb9-9fcd-34b26892d29a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-788e42-22pnn\" (UID: \"29f6a42e-027c-4fb9-9fcd-34b26892d29a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-788e42-22pnn" Oct 30 22:14:50.433857 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:50.433755 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjzdw\" (UniqueName: \"kubernetes.io/projected/29f6a42e-027c-4fb9-9fcd-34b26892d29a-kube-api-access-pjzdw\") pod \"maas-api-revoke-keys-e2e-worker-single-main-788e42-22pnn\" (UID: \"29f6a42e-027c-4fb9-9fcd-34b26892d29a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-788e42-22pnn" Oct 30 22:14:50.434362 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:50.434323 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29f6a42e-027c-4fb9-9fcd-34b26892d29a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-788e42-22pnn\" (UID: \"29f6a42e-027c-4fb9-9fcd-34b26892d29a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-788e42-22pnn" Oct 30 22:14:50.436101 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:50.436076 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29f6a42e-027c-4fb9-9fcd-34b26892d29a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-788e42-22pnn\" (UID: \"29f6a42e-027c-4fb9-9fcd-34b26892d29a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-788e42-22pnn" Oct 30 22:14:50.491987 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:50.491962 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjzdw\" (UniqueName: \"kubernetes.io/projected/29f6a42e-027c-4fb9-9fcd-34b26892d29a-kube-api-access-pjzdw\") pod \"maas-api-revoke-keys-e2e-worker-single-main-788e42-22pnn\" (UID: \"29f6a42e-027c-4fb9-9fcd-34b26892d29a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-788e42-22pnn" Oct 30 22:14:50.780936 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:50.780908 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-788e42-22pnn" Oct 30 22:14:51.151091 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:51.151006 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-788e42-22pnn"] Oct 30 22:14:51.157618 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:14:51.157591 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29f6a42e_027c_4fb9_9fcd_34b26892d29a.slice/crio-7d9edf7de40467d909051c3ea000047c4c9198ecbefb10b3986a412a292c2021 WatchSource:0}: Error finding container 7d9edf7de40467d909051c3ea000047c4c9198ecbefb10b3986a412a292c2021: Status 404 returned error can't find the container with id 7d9edf7de40467d909051c3ea000047c4c9198ecbefb10b3986a412a292c2021 Oct 30 22:14:52.093617 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:52.093582 4560 generic.go:356] "Generic (PLEG): container finished" podID="29f6a42e-027c-4fb9-9fcd-34b26892d29a" containerID="f316e80b256afd22413784ee5f84967929f9f14a476839b77b7d84e3b6533227" exitCode=0 Oct 30 22:14:52.093972 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:52.093660 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-788e42-22pnn" event={"ID":"29f6a42e-027c-4fb9-9fcd-34b26892d29a","Type":"ContainerDied","Data":"f316e80b256afd22413784ee5f84967929f9f14a476839b77b7d84e3b6533227"} Oct 30 22:14:52.093972 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:52.093711 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-788e42-22pnn" event={"ID":"29f6a42e-027c-4fb9-9fcd-34b26892d29a","Type":"ContainerStarted","Data":"7d9edf7de40467d909051c3ea000047c4c9198ecbefb10b3986a412a292c2021"} Oct 30 22:14:52.891660 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:52.891628 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:14:53.219790 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:53.219766 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-788e42-22pnn" Oct 30 22:14:53.361409 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:53.361376 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29f6a42e-027c-4fb9-9fcd-34b26892d29a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29f6a42e-027c-4fb9-9fcd-34b26892d29a-maas-api-xks-ca\") pod \"29f6a42e-027c-4fb9-9fcd-34b26892d29a\" (UID: \"29f6a42e-027c-4fb9-9fcd-34b26892d29a\") " Oct 30 22:14:53.361576 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:53.361465 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29f6a42e-027c-4fb9-9fcd-34b26892d29a-kube-api-access-pjzdw\" (UniqueName: \"kubernetes.io/projected/29f6a42e-027c-4fb9-9fcd-34b26892d29a-kube-api-access-pjzdw\") pod \"29f6a42e-027c-4fb9-9fcd-34b26892d29a\" (UID: \"29f6a42e-027c-4fb9-9fcd-34b26892d29a\") " Oct 30 22:14:53.361641 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:53.361582 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29f6a42e-027c-4fb9-9fcd-34b26892d29a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29f6a42e-027c-4fb9-9fcd-34b26892d29a-maas-api-openshift-ca\") pod \"29f6a42e-027c-4fb9-9fcd-34b26892d29a\" (UID: \"29f6a42e-027c-4fb9-9fcd-34b26892d29a\") " Oct 30 22:14:53.361929 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:53.361902 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29f6a42e-027c-4fb9-9fcd-34b26892d29a-maas-api-openshift-ca" pod "29f6a42e-027c-4fb9-9fcd-34b26892d29a" (UID: "29f6a42e-027c-4fb9-9fcd-34b26892d29a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:53.363569 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:53.363547 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29f6a42e-027c-4fb9-9fcd-34b26892d29a-kube-api-access-pjzdw" pod "29f6a42e-027c-4fb9-9fcd-34b26892d29a" (UID: "29f6a42e-027c-4fb9-9fcd-34b26892d29a"). InnerVolumeSpecName "kube-api-access-pjzdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:53.363814 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:53.363791 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29f6a42e-027c-4fb9-9fcd-34b26892d29a-maas-api-xks-ca" pod "29f6a42e-027c-4fb9-9fcd-34b26892d29a" (UID: "29f6a42e-027c-4fb9-9fcd-34b26892d29a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:53.462732 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:53.462662 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29f6a42e-027c-4fb9-9fcd-34b26892d29a-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:53.462732 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:53.462689 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjzdw\" (UniqueName: \"kubernetes.io/projected/29f6a42e-027c-4fb9-9fcd-34b26892d29a-kube-api-access-pjzdw\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:53.462732 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:53.462700 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29f6a42e-027c-4fb9-9fcd-34b26892d29a-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:54.102143 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:54.102112 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-788e42-22pnn" event={"ID":"29f6a42e-027c-4fb9-9fcd-34b26892d29a","Type":"ContainerDied","Data":"7d9edf7de40467d909051c3ea000047c4c9198ecbefb10b3986a412a292c2021"} Oct 30 22:14:54.102143 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:54.102144 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-788e42-22pnn" Oct 30 22:14:54.102375 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:14:54.102158 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d9edf7de40467d909051c3ea000047c4c9198ecbefb10b3986a412a292c2021" Oct 30 22:15:00.192664 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:00.192629 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-788e42-29889978mdtf"] Oct 30 22:15:00.193062 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:00.192935 4560 cpu_manager.go:414] "containerMap: removing container" podUID="29f6a42e-027c-4fb9-9fcd-34b26892d29a" containerName="revoke-keys" Oct 30 22:15:00.193062 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:00.192945 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29f6a42e-027c-4fb9-9fcd-34b26892d29a" containerName="revoke-keys" Oct 30 22:15:00.193062 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:00.193006 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="29f6a42e-027c-4fb9-9fcd-34b26892d29a" containerName="revoke-keys" Oct 30 22:15:00.196069 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:00.196042 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-788e42-29889978mdtf" Oct 30 22:15:00.251356 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:00.251315 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:15:00.297463 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:00.297435 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:15:00.297463 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:00.297435 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:15:00.300480 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:00.300457 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-788e42-29889978mdtf"] Oct 30 22:15:00.320539 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:00.320510 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-788e42-29889978mdtf\" (UID: \"e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-788e42-29889978mdtf" Oct 30 22:15:00.320662 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:00.320572 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxl46\" (UniqueName: \"kubernetes.io/projected/e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c-kube-api-access-vxl46\") pod \"maas-api-key-cleanup-e2e-worker-single-main-788e42-29889978mdtf\" (UID: \"e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-788e42-29889978mdtf" Oct 30 22:15:00.421687 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:00.421661 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-788e42-29889978mdtf\" (UID: \"e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-788e42-29889978mdtf" Oct 30 22:15:00.421819 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:00.421703 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxl46\" (UniqueName: \"kubernetes.io/projected/e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c-kube-api-access-vxl46\") pod \"maas-api-key-cleanup-e2e-worker-single-main-788e42-29889978mdtf\" (UID: \"e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-788e42-29889978mdtf" Oct 30 22:15:00.422309 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:00.422291 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-788e42-29889978mdtf\" (UID: \"e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-788e42-29889978mdtf" Oct 30 22:15:00.480424 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:00.480396 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxl46\" (UniqueName: \"kubernetes.io/projected/e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c-kube-api-access-vxl46\") pod \"maas-api-key-cleanup-e2e-worker-single-main-788e42-29889978mdtf\" (UID: \"e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-788e42-29889978mdtf" Oct 30 22:15:00.509429 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:00.509407 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-788e42-29889978mdtf" Oct 30 22:15:00.889655 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:00.889578 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-788e42-29889978mdtf"] Oct 30 22:15:00.896077 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:15:00.896044 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5dd8a4d_82ac_45c6_9c5e_32b4bce95d4c.slice/crio-97a4328ac2facceb3b4a284e67bd93b1712a97793085e03054a14a2578c3adb7 WatchSource:0}: Error finding container 97a4328ac2facceb3b4a284e67bd93b1712a97793085e03054a14a2578c3adb7: Status 404 returned error can't find the container with id 97a4328ac2facceb3b4a284e67bd93b1712a97793085e03054a14a2578c3adb7 Oct 30 22:15:01.131927 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:01.131888 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-788e42-29889978mdtf" event={"ID":"e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c","Type":"ContainerStarted","Data":"97a4328ac2facceb3b4a284e67bd93b1712a97793085e03054a14a2578c3adb7"} Oct 30 22:15:01.617618 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:01.615276 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-788e42-29889978mdtf"] Oct 30 22:15:02.137171 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:02.137133 4560 generic.go:356] "Generic (PLEG): container finished" podID="e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c" containerID="5d73c1f2b25e3cae147bc3083585fe8d3f919c7e7e074be383e94f528c1fcd57" exitCode=6 Oct 30 22:15:02.137332 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:02.137212 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-788e42-29889978mdtf" event={"ID":"e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c","Type":"ContainerDied","Data":"5d73c1f2b25e3cae147bc3083585fe8d3f919c7e7e074be383e94f528c1fcd57"} Oct 30 22:15:02.137332 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:02.137314 4560 scope.go:122] "RemoveContainer" containerID="5d73c1f2b25e3cae147bc3083585fe8d3f919c7e7e074be383e94f528c1fcd57" Oct 30 22:15:03.142983 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:03.142890 4560 generic.go:356] "Generic (PLEG): container finished" podID="e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c" containerID="943750fa8869921bd47a4cc223527853cbc84568ada4c29667f8a2e66f714eac" exitCode=6 Oct 30 22:15:03.142983 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:03.142953 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-788e42-29889978mdtf" event={"ID":"e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c","Type":"ContainerDied","Data":"943750fa8869921bd47a4cc223527853cbc84568ada4c29667f8a2e66f714eac"} Oct 30 22:15:03.143507 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:03.143008 4560 scope.go:122] "RemoveContainer" containerID="5d73c1f2b25e3cae147bc3083585fe8d3f919c7e7e074be383e94f528c1fcd57" Oct 30 22:15:03.276306 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:03.276282 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-788e42-29889978mdtf" Oct 30 22:15:03.348689 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:03.348656 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c-kube-api-access-vxl46\" (UniqueName: \"kubernetes.io/projected/e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c-kube-api-access-vxl46\") pod \"e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c\" (UID: \"e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c\") " Oct 30 22:15:03.348689 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:03.348703 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c-maas-api-service-ca\") pod \"e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c\" (UID: \"e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c\") " Oct 30 22:15:03.349116 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:03.349088 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c-maas-api-service-ca" pod "e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c" (UID: "e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:03.350841 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:03.350818 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c-kube-api-access-vxl46" pod "e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c" (UID: "e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c"). InnerVolumeSpecName "kube-api-access-vxl46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:03.449329 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:03.449239 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxl46\" (UniqueName: \"kubernetes.io/projected/e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c-kube-api-access-vxl46\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:03.449329 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:03.449270 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c-maas-api-service-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:04.148398 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:04.148368 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-788e42-29889978mdtf" Oct 30 22:15:04.148398 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:04.148379 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-788e42-29889978mdtf" event={"ID":"e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c","Type":"ContainerDied","Data":"97a4328ac2facceb3b4a284e67bd93b1712a97793085e03054a14a2578c3adb7"} Oct 30 22:15:04.148901 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:04.148438 4560 scope.go:122] "RemoveContainer" containerID="943750fa8869921bd47a4cc223527853cbc84568ada4c29667f8a2e66f714eac" Oct 30 22:15:04.303731 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:04.303696 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-788e42-29889978mdtf"] Oct 30 22:15:04.350560 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:04.350520 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-788e42-29889978mdtf"] Oct 30 22:15:04.574239 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:04.574207 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c" path="/var/lib/kubelet/pods/e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c/volumes" Oct 30 22:15:06.840299 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:06.840260 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-788e42-22pnn"] Oct 30 22:15:06.885567 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:06.885536 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-788e42-22pnn"] Oct 30 22:15:08.574285 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:08.574250 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29f6a42e-027c-4fb9-9fcd-34b26892d29a" path="/var/lib/kubelet/pods/29f6a42e-027c-4fb9-9fcd-34b26892d29a/volumes" Oct 30 22:15:09.308423 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:09.308387 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:15:19.683450 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:15:19.683412 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:15:30.793025 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:30.792991 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:15:32.368506 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:32.368451 4560 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-m2sxq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:15:32.369159 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:32.369109 4560 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-m2sxq" podUID="c73ef128-5710-4654-a155-4b9083ab04bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:41.763248 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:15:41.763212 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:15:49.682997 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:15:49.682954 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:16:19.683968 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:16:19.683884 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:16:49.683392 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:16:49.683354 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:17:15.514078 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:17:15.514046 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:17:19.683812 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:17:19.683770 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:17:33.053730 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:17:33.053654 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:17:40.908719 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:17:40.908685 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:17:49.683417 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:17:49.683376 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:18:02.302099 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:18:02.302060 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:18:19.683766 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:18:19.683729 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:18:21.921940 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:18:21.921905 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:18:40.565142 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:18:40.565100 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:18:49.704964 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:18:49.704920 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:19:00.717989 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:00.717906 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:19:15.755391 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:15.755353 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c5118f8-9kzcp"] Oct 30 22:19:15.755871 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:15.755651 4560 cpu_manager.go:414] "containerMap: removing container" podUID="e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c" containerName="cleanup" Oct 30 22:19:15.755871 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:15.755660 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c" containerName="cleanup" Oct 30 22:19:15.755871 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:15.755674 4560 cpu_manager.go:414] "containerMap: removing container" podUID="e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c" containerName="cleanup" Oct 30 22:19:15.755871 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:15.755680 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c" containerName="cleanup" Oct 30 22:19:15.755871 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:15.755741 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c" containerName="cleanup" Oct 30 22:19:15.758544 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:15.758524 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5118f8-9kzcp" Oct 30 22:19:15.776354 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:15.776315 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c40445ab-df13-4409-b048-b5baea2586e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c5118f8-9kzcp\" (UID: \"c40445ab-df13-4409-b048-b5baea2586e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5118f8-9kzcp" Oct 30 22:19:15.776464 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:15.776377 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5gh4\" (UniqueName: \"kubernetes.io/projected/c40445ab-df13-4409-b048-b5baea2586e9-kube-api-access-n5gh4\") pod \"maas-api-revoke-sub-0c5118f8-9kzcp\" (UID: \"c40445ab-df13-4409-b048-b5baea2586e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5118f8-9kzcp" Oct 30 22:19:15.776464 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:15.776449 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c40445ab-df13-4409-b048-b5baea2586e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c5118f8-9kzcp\" (UID: \"c40445ab-df13-4409-b048-b5baea2586e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5118f8-9kzcp" Oct 30 22:19:15.854689 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:15.854657 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:19:15.854689 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:15.854675 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:19:15.854689 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:15.854675 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:19:15.854970 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:15.854801 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:19:15.856178 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:15.856161 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c5118f8-9kzcp"] Oct 30 22:19:15.877259 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:15.877236 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c40445ab-df13-4409-b048-b5baea2586e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c5118f8-9kzcp\" (UID: \"c40445ab-df13-4409-b048-b5baea2586e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5118f8-9kzcp" Oct 30 22:19:15.877403 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:15.877275 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5gh4\" (UniqueName: \"kubernetes.io/projected/c40445ab-df13-4409-b048-b5baea2586e9-kube-api-access-n5gh4\") pod \"maas-api-revoke-sub-0c5118f8-9kzcp\" (UID: \"c40445ab-df13-4409-b048-b5baea2586e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5118f8-9kzcp" Oct 30 22:19:15.877403 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:15.877293 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c40445ab-df13-4409-b048-b5baea2586e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c5118f8-9kzcp\" (UID: \"c40445ab-df13-4409-b048-b5baea2586e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5118f8-9kzcp" Oct 30 22:19:15.877958 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:15.877938 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c40445ab-df13-4409-b048-b5baea2586e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c5118f8-9kzcp\" (UID: \"c40445ab-df13-4409-b048-b5baea2586e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5118f8-9kzcp" Oct 30 22:19:15.879695 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:15.879673 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c40445ab-df13-4409-b048-b5baea2586e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c5118f8-9kzcp\" (UID: \"c40445ab-df13-4409-b048-b5baea2586e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5118f8-9kzcp" Oct 30 22:19:15.936092 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:15.936066 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5gh4\" (UniqueName: \"kubernetes.io/projected/c40445ab-df13-4409-b048-b5baea2586e9-kube-api-access-n5gh4\") pod \"maas-api-revoke-sub-0c5118f8-9kzcp\" (UID: \"c40445ab-df13-4409-b048-b5baea2586e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5118f8-9kzcp" Oct 30 22:19:16.071540 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:16.071444 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5118f8-9kzcp" Oct 30 22:19:16.448572 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:16.448453 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c5118f8-9kzcp"] Oct 30 22:19:16.456247 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:19:16.456220 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc40445ab_df13_4409_b048_b5baea2586e9.slice/crio-75f6f17657e550692194a3905c29dd49e9573751c73222012245d7c7b6c416ee WatchSource:0}: Error finding container 75f6f17657e550692194a3905c29dd49e9573751c73222012245d7c7b6c416ee: Status 404 returned error can't find the container with id 75f6f17657e550692194a3905c29dd49e9573751c73222012245d7c7b6c416ee Oct 30 22:19:17.184972 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:17.184883 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5118f8-9kzcp" event={"ID":"c40445ab-df13-4409-b048-b5baea2586e9","Type":"ContainerStarted","Data":"600f51599bd098bd5861220d7161d1b1e640b597d648d502c2750fb55a93d134"} Oct 30 22:19:17.184972 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:17.184937 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5118f8-9kzcp" event={"ID":"c40445ab-df13-4409-b048-b5baea2586e9","Type":"ContainerStarted","Data":"75f6f17657e550692194a3905c29dd49e9573751c73222012245d7c7b6c416ee"} Oct 30 22:19:18.190527 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:18.190491 4560 generic.go:356] "Generic (PLEG): container finished" podID="c40445ab-df13-4409-b048-b5baea2586e9" containerID="600f51599bd098bd5861220d7161d1b1e640b597d648d502c2750fb55a93d134" exitCode=0 Oct 30 22:19:18.190932 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:18.190580 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5118f8-9kzcp" event={"ID":"c40445ab-df13-4409-b048-b5baea2586e9","Type":"ContainerDied","Data":"600f51599bd098bd5861220d7161d1b1e640b597d648d502c2750fb55a93d134"} Oct 30 22:19:19.326701 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:19.326678 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5118f8-9kzcp" Oct 30 22:19:19.406551 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:19.406519 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c40445ab-df13-4409-b048-b5baea2586e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c40445ab-df13-4409-b048-b5baea2586e9-maas-api-openshift-ca\") pod \"c40445ab-df13-4409-b048-b5baea2586e9\" (UID: \"c40445ab-df13-4409-b048-b5baea2586e9\") " Oct 30 22:19:19.406724 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:19.406565 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c40445ab-df13-4409-b048-b5baea2586e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c40445ab-df13-4409-b048-b5baea2586e9-maas-api-xks-ca\") pod \"c40445ab-df13-4409-b048-b5baea2586e9\" (UID: \"c40445ab-df13-4409-b048-b5baea2586e9\") " Oct 30 22:19:19.406724 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:19.406649 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c40445ab-df13-4409-b048-b5baea2586e9-kube-api-access-n5gh4\" (UniqueName: \"kubernetes.io/projected/c40445ab-df13-4409-b048-b5baea2586e9-kube-api-access-n5gh4\") pod \"c40445ab-df13-4409-b048-b5baea2586e9\" (UID: \"c40445ab-df13-4409-b048-b5baea2586e9\") " Oct 30 22:19:19.406926 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:19.406903 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c40445ab-df13-4409-b048-b5baea2586e9-maas-api-openshift-ca" pod "c40445ab-df13-4409-b048-b5baea2586e9" (UID: "c40445ab-df13-4409-b048-b5baea2586e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:19:19.408682 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:19.408661 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c40445ab-df13-4409-b048-b5baea2586e9-kube-api-access-n5gh4" pod "c40445ab-df13-4409-b048-b5baea2586e9" (UID: "c40445ab-df13-4409-b048-b5baea2586e9"). InnerVolumeSpecName "kube-api-access-n5gh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:19.408682 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:19.408668 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c40445ab-df13-4409-b048-b5baea2586e9-maas-api-xks-ca" pod "c40445ab-df13-4409-b048-b5baea2586e9" (UID: "c40445ab-df13-4409-b048-b5baea2586e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:19:19.507405 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:19.507365 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5gh4\" (UniqueName: \"kubernetes.io/projected/c40445ab-df13-4409-b048-b5baea2586e9-kube-api-access-n5gh4\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:19.507405 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:19.507404 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c40445ab-df13-4409-b048-b5baea2586e9-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:19.507405 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:19.507415 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c40445ab-df13-4409-b048-b5baea2586e9-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:19.691088 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:19:19.691052 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:19:20.198961 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:20.198928 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5118f8-9kzcp" event={"ID":"c40445ab-df13-4409-b048-b5baea2586e9","Type":"ContainerDied","Data":"75f6f17657e550692194a3905c29dd49e9573751c73222012245d7c7b6c416ee"} Oct 30 22:19:20.198961 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:20.198959 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5118f8-9kzcp" Oct 30 22:19:20.199202 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:20.198979 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75f6f17657e550692194a3905c29dd49e9573751c73222012245d7c7b6c416ee" Oct 30 22:19:27.467835 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:27.467800 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ec071-nctpf"] Oct 30 22:19:27.468317 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:27.468095 4560 cpu_manager.go:414] "containerMap: removing container" podUID="c40445ab-df13-4409-b048-b5baea2586e9" containerName="revoke-keys" Oct 30 22:19:27.468317 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:27.468106 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c40445ab-df13-4409-b048-b5baea2586e9" containerName="revoke-keys" Oct 30 22:19:27.468317 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:27.468188 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="c40445ab-df13-4409-b048-b5baea2586e9" containerName="revoke-keys" Oct 30 22:19:27.468317 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:27.468197 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5dd8a4d-82ac-45c6-9c5e-32b4bce95d4c" containerName="cleanup" Oct 30 22:19:27.471083 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:27.471067 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ec071-nctpf" Oct 30 22:19:27.520679 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:27.520654 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:19:27.567619 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:27.567582 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:19:27.567619 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:27.567591 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:19:27.567619 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:27.567591 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6npn\"" Oct 30 22:19:27.569110 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:27.569085 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ec071-nctpf"] Oct 30 22:19:27.572298 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:27.572277 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s4ft\" (UniqueName: \"kubernetes.io/projected/6f81a339-f925-4685-b410-31aec29e0ec7-kube-api-access-8s4ft\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3ec071-nctpf\" (UID: \"6f81a339-f925-4685-b410-31aec29e0ec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ec071-nctpf" Oct 30 22:19:27.572412 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:27.572332 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f81a339-f925-4685-b410-31aec29e0ec7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3ec071-nctpf\" (UID: \"6f81a339-f925-4685-b410-31aec29e0ec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ec071-nctpf" Oct 30 22:19:27.572462 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:27.572430 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f81a339-f925-4685-b410-31aec29e0ec7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3ec071-nctpf\" (UID: \"6f81a339-f925-4685-b410-31aec29e0ec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ec071-nctpf" Oct 30 22:19:27.673148 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:27.673109 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f81a339-f925-4685-b410-31aec29e0ec7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3ec071-nctpf\" (UID: \"6f81a339-f925-4685-b410-31aec29e0ec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ec071-nctpf" Oct 30 22:19:27.673329 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:27.673165 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s4ft\" (UniqueName: \"kubernetes.io/projected/6f81a339-f925-4685-b410-31aec29e0ec7-kube-api-access-8s4ft\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3ec071-nctpf\" (UID: \"6f81a339-f925-4685-b410-31aec29e0ec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ec071-nctpf" Oct 30 22:19:27.673329 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:27.673253 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f81a339-f925-4685-b410-31aec29e0ec7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3ec071-nctpf\" (UID: \"6f81a339-f925-4685-b410-31aec29e0ec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ec071-nctpf" Oct 30 22:19:27.673963 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:27.673930 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f81a339-f925-4685-b410-31aec29e0ec7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3ec071-nctpf\" (UID: \"6f81a339-f925-4685-b410-31aec29e0ec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ec071-nctpf" Oct 30 22:19:27.675487 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:27.675466 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f81a339-f925-4685-b410-31aec29e0ec7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3ec071-nctpf\" (UID: \"6f81a339-f925-4685-b410-31aec29e0ec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ec071-nctpf" Oct 30 22:19:27.731464 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:27.731437 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s4ft\" (UniqueName: \"kubernetes.io/projected/6f81a339-f925-4685-b410-31aec29e0ec7-kube-api-access-8s4ft\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3ec071-nctpf\" (UID: \"6f81a339-f925-4685-b410-31aec29e0ec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ec071-nctpf" Oct 30 22:19:27.783276 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:27.783249 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ec071-nctpf" Oct 30 22:19:28.191708 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:28.191627 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ec071-nctpf"] Oct 30 22:19:28.198640 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:19:28.198611 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f81a339_f925_4685_b410_31aec29e0ec7.slice/crio-8944c718afab6ce6cdc05577cb2ca9ef29b7471edeeeb707f4f67ca892b1aa4a WatchSource:0}: Error finding container 8944c718afab6ce6cdc05577cb2ca9ef29b7471edeeeb707f4f67ca892b1aa4a: Status 404 returned error can't find the container with id 8944c718afab6ce6cdc05577cb2ca9ef29b7471edeeeb707f4f67ca892b1aa4a Oct 30 22:19:28.236964 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:28.236937 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ec071-nctpf" event={"ID":"6f81a339-f925-4685-b410-31aec29e0ec7","Type":"ContainerStarted","Data":"8944c718afab6ce6cdc05577cb2ca9ef29b7471edeeeb707f4f67ca892b1aa4a"} Oct 30 22:19:28.773965 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:28.773937 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:19:29.242518 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:29.242481 4560 generic.go:356] "Generic (PLEG): container finished" podID="6f81a339-f925-4685-b410-31aec29e0ec7" containerID="8b01b6961d6d5ca87a41fdf7ccc672bce3ee9fb10c263e1a821a2c64212d4391" exitCode=0 Oct 30 22:19:29.242707 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:29.242554 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ec071-nctpf" event={"ID":"6f81a339-f925-4685-b410-31aec29e0ec7","Type":"ContainerDied","Data":"8b01b6961d6d5ca87a41fdf7ccc672bce3ee9fb10c263e1a821a2c64212d4391"} Oct 30 22:19:30.370230 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:30.370207 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ec071-nctpf" Oct 30 22:19:30.498596 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:30.498513 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f81a339-f925-4685-b410-31aec29e0ec7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f81a339-f925-4685-b410-31aec29e0ec7-maas-api-openshift-ca\") pod \"6f81a339-f925-4685-b410-31aec29e0ec7\" (UID: \"6f81a339-f925-4685-b410-31aec29e0ec7\") " Oct 30 22:19:30.498596 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:30.498574 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f81a339-f925-4685-b410-31aec29e0ec7-kube-api-access-8s4ft\" (UniqueName: \"kubernetes.io/projected/6f81a339-f925-4685-b410-31aec29e0ec7-kube-api-access-8s4ft\") pod \"6f81a339-f925-4685-b410-31aec29e0ec7\" (UID: \"6f81a339-f925-4685-b410-31aec29e0ec7\") " Oct 30 22:19:30.498596 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:30.498597 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f81a339-f925-4685-b410-31aec29e0ec7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f81a339-f925-4685-b410-31aec29e0ec7-maas-api-xks-ca\") pod \"6f81a339-f925-4685-b410-31aec29e0ec7\" (UID: \"6f81a339-f925-4685-b410-31aec29e0ec7\") " Oct 30 22:19:30.498912 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:30.498857 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f81a339-f925-4685-b410-31aec29e0ec7-maas-api-openshift-ca" pod "6f81a339-f925-4685-b410-31aec29e0ec7" (UID: "6f81a339-f925-4685-b410-31aec29e0ec7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:19:30.500746 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:30.500713 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f81a339-f925-4685-b410-31aec29e0ec7-maas-api-xks-ca" pod "6f81a339-f925-4685-b410-31aec29e0ec7" (UID: "6f81a339-f925-4685-b410-31aec29e0ec7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:19:30.500862 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:30.500754 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f81a339-f925-4685-b410-31aec29e0ec7-kube-api-access-8s4ft" pod "6f81a339-f925-4685-b410-31aec29e0ec7" (UID: "6f81a339-f925-4685-b410-31aec29e0ec7"). InnerVolumeSpecName "kube-api-access-8s4ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:30.599888 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:30.599861 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f81a339-f925-4685-b410-31aec29e0ec7-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:30.599888 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:30.599887 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s4ft\" (UniqueName: \"kubernetes.io/projected/6f81a339-f925-4685-b410-31aec29e0ec7-kube-api-access-8s4ft\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:30.600062 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:30.599896 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f81a339-f925-4685-b410-31aec29e0ec7-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:31.257871 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:31.257838 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ec071-nctpf" Oct 30 22:19:31.257871 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:31.257853 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ec071-nctpf" event={"ID":"6f81a339-f925-4685-b410-31aec29e0ec7","Type":"ContainerDied","Data":"8944c718afab6ce6cdc05577cb2ca9ef29b7471edeeeb707f4f67ca892b1aa4a"} Oct 30 22:19:31.258091 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:31.257930 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8944c718afab6ce6cdc05577cb2ca9ef29b7471edeeeb707f4f67ca892b1aa4a" Oct 30 22:19:42.045726 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:42.045690 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14e189aa-mgdk7"] Oct 30 22:19:42.092527 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:42.092488 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14e189aa-mgdk7"] Oct 30 22:19:42.573947 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:42.573910 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e212d07-7722-4ba5-ae46-e4f72946c0f0" path="/var/lib/kubelet/pods/0e212d07-7722-4ba5-ae46-e4f72946c0f0/volumes" Oct 30 22:19:46.421287 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:46.421256 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ec071-nctpf"] Oct 30 22:19:46.468049 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:46.468016 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ec071-nctpf"] Oct 30 22:19:46.573749 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:46.573715 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f81a339-f925-4685-b410-31aec29e0ec7" path="/var/lib/kubelet/pods/6f81a339-f925-4685-b410-31aec29e0ec7/volumes" Oct 30 22:19:49.126175 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:19:49.126136 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:19:49.684037 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:19:49.683994 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:20:04.816602 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:20:04.816566 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:20:19.683394 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:20:19.683354 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:20:47.871523 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:20:47.871484 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:20:48.135865 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:20:48.135777 4560 scope.go:122] "RemoveContainer" containerID="9098dc42dac17882eb646998401c3b0356794d0363e495b30cb34f37d1eed4cb" Oct 30 22:20:49.719523 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:20:49.719481 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:21:18.979953 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:21:18.979917 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:21:19.695472 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:21:19.695420 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:21:34.589506 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:21:34.589469 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:21:48.155712 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:21:48.155682 4560 scope.go:122] "RemoveContainer" containerID="f316e80b256afd22413784ee5f84967929f9f14a476839b77b7d84e3b6533227" Oct 30 22:21:49.683460 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:21:49.683425 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:21:53.507271 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:21:53.507237 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:22:05.860421 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:22:05.860332 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:22:17.166838 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:22:17.166803 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:22:19.683912 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:22:19.683876 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:22:39.859087 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:22:39.859050 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:22:49.684133 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:22:49.684088 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:23:04.695476 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:04.695438 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:23:19.683766 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:23:19.683728 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:23:27.487503 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:27.487420 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c501563-7lfp4"] Oct 30 22:23:27.487991 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:27.487725 4560 cpu_manager.go:414] "containerMap: removing container" podUID="6f81a339-f925-4685-b410-31aec29e0ec7" containerName="revoke-keys" Oct 30 22:23:27.487991 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:27.487736 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f81a339-f925-4685-b410-31aec29e0ec7" containerName="revoke-keys" Oct 30 22:23:27.487991 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:27.487801 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f81a339-f925-4685-b410-31aec29e0ec7" containerName="revoke-keys" Oct 30 22:23:27.490637 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:27.490618 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c501563-7lfp4" Oct 30 22:23:27.559452 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:27.559407 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85a4b13-e50d-40e5-91a4-64af9a411765-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c501563-7lfp4\" (UID: \"d85a4b13-e50d-40e5-91a4-64af9a411765\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c501563-7lfp4" Oct 30 22:23:27.559648 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:27.559532 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65tsx\" (UniqueName: \"kubernetes.io/projected/d85a4b13-e50d-40e5-91a4-64af9a411765-kube-api-access-65tsx\") pod \"maas-api-revoke-keys-e2e-ait-8c501563-7lfp4\" (UID: \"d85a4b13-e50d-40e5-91a4-64af9a411765\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c501563-7lfp4" Oct 30 22:23:27.559648 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:27.559623 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85a4b13-e50d-40e5-91a4-64af9a411765-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c501563-7lfp4\" (UID: \"d85a4b13-e50d-40e5-91a4-64af9a411765\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c501563-7lfp4" Oct 30 22:23:27.586995 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:27.586964 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:23:27.586995 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:27.586967 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:23:27.587223 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:27.586988 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:23:27.587223 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:27.587018 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7b9qb\"" Oct 30 22:23:27.588411 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:27.588389 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c501563-7lfp4"] Oct 30 22:23:27.660304 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:27.660271 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85a4b13-e50d-40e5-91a4-64af9a411765-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c501563-7lfp4\" (UID: \"d85a4b13-e50d-40e5-91a4-64af9a411765\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c501563-7lfp4" Oct 30 22:23:27.660484 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:27.660321 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65tsx\" (UniqueName: \"kubernetes.io/projected/d85a4b13-e50d-40e5-91a4-64af9a411765-kube-api-access-65tsx\") pod \"maas-api-revoke-keys-e2e-ait-8c501563-7lfp4\" (UID: \"d85a4b13-e50d-40e5-91a4-64af9a411765\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c501563-7lfp4" Oct 30 22:23:27.660484 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:27.660456 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85a4b13-e50d-40e5-91a4-64af9a411765-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c501563-7lfp4\" (UID: \"d85a4b13-e50d-40e5-91a4-64af9a411765\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c501563-7lfp4" Oct 30 22:23:27.661019 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:27.660996 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85a4b13-e50d-40e5-91a4-64af9a411765-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c501563-7lfp4\" (UID: \"d85a4b13-e50d-40e5-91a4-64af9a411765\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c501563-7lfp4" Oct 30 22:23:27.662850 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:27.662830 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85a4b13-e50d-40e5-91a4-64af9a411765-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c501563-7lfp4\" (UID: \"d85a4b13-e50d-40e5-91a4-64af9a411765\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c501563-7lfp4" Oct 30 22:23:27.718589 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:27.718566 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65tsx\" (UniqueName: \"kubernetes.io/projected/d85a4b13-e50d-40e5-91a4-64af9a411765-kube-api-access-65tsx\") pod \"maas-api-revoke-keys-e2e-ait-8c501563-7lfp4\" (UID: \"d85a4b13-e50d-40e5-91a4-64af9a411765\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c501563-7lfp4" Oct 30 22:23:27.802984 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:27.802906 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c501563-7lfp4" Oct 30 22:23:28.175190 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:28.175094 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c501563-7lfp4"] Oct 30 22:23:28.181966 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:23:28.181938 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd85a4b13_e50d_40e5_91a4_64af9a411765.slice/crio-144df70f8b130ecc5e563d3f0691af7f069811f7f91a0f063d28b37ede707cc3 WatchSource:0}: Error finding container 144df70f8b130ecc5e563d3f0691af7f069811f7f91a0f063d28b37ede707cc3: Status 404 returned error can't find the container with id 144df70f8b130ecc5e563d3f0691af7f069811f7f91a0f063d28b37ede707cc3 Oct 30 22:23:28.312400 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:28.312365 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c501563-7lfp4" event={"ID":"d85a4b13-e50d-40e5-91a4-64af9a411765","Type":"ContainerStarted","Data":"144df70f8b130ecc5e563d3f0691af7f069811f7f91a0f063d28b37ede707cc3"} Oct 30 22:23:29.318126 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:29.318036 4560 generic.go:356] "Generic (PLEG): container finished" podID="d85a4b13-e50d-40e5-91a4-64af9a411765" containerID="b7f45670f7488509abe1a322e97052ccefb7382299d0380f620dd60c5cbf459d" exitCode=0 Oct 30 22:23:29.318126 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:29.318086 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c501563-7lfp4" event={"ID":"d85a4b13-e50d-40e5-91a4-64af9a411765","Type":"ContainerDied","Data":"b7f45670f7488509abe1a322e97052ccefb7382299d0380f620dd60c5cbf459d"} Oct 30 22:23:30.446118 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:30.446086 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c501563-7lfp4" Oct 30 22:23:30.593236 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:30.593168 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d85a4b13-e50d-40e5-91a4-64af9a411765-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85a4b13-e50d-40e5-91a4-64af9a411765-maas-api-xks-ca\") pod \"d85a4b13-e50d-40e5-91a4-64af9a411765\" (UID: \"d85a4b13-e50d-40e5-91a4-64af9a411765\") " Oct 30 22:23:30.593415 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:30.593252 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d85a4b13-e50d-40e5-91a4-64af9a411765-kube-api-access-65tsx\" (UniqueName: \"kubernetes.io/projected/d85a4b13-e50d-40e5-91a4-64af9a411765-kube-api-access-65tsx\") pod \"d85a4b13-e50d-40e5-91a4-64af9a411765\" (UID: \"d85a4b13-e50d-40e5-91a4-64af9a411765\") " Oct 30 22:23:30.593415 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:30.593311 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d85a4b13-e50d-40e5-91a4-64af9a411765-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85a4b13-e50d-40e5-91a4-64af9a411765-maas-api-openshift-ca\") pod \"d85a4b13-e50d-40e5-91a4-64af9a411765\" (UID: \"d85a4b13-e50d-40e5-91a4-64af9a411765\") " Oct 30 22:23:30.593709 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:30.593674 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d85a4b13-e50d-40e5-91a4-64af9a411765-maas-api-openshift-ca" pod "d85a4b13-e50d-40e5-91a4-64af9a411765" (UID: "d85a4b13-e50d-40e5-91a4-64af9a411765"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:30.595285 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:30.595263 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d85a4b13-e50d-40e5-91a4-64af9a411765-kube-api-access-65tsx" pod "d85a4b13-e50d-40e5-91a4-64af9a411765" (UID: "d85a4b13-e50d-40e5-91a4-64af9a411765"). InnerVolumeSpecName "kube-api-access-65tsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:30.595391 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:30.595267 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d85a4b13-e50d-40e5-91a4-64af9a411765-maas-api-xks-ca" pod "d85a4b13-e50d-40e5-91a4-64af9a411765" (UID: "d85a4b13-e50d-40e5-91a4-64af9a411765"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:23:30.694409 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:30.694379 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65tsx\" (UniqueName: \"kubernetes.io/projected/d85a4b13-e50d-40e5-91a4-64af9a411765-kube-api-access-65tsx\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:30.694409 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:30.694405 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85a4b13-e50d-40e5-91a4-64af9a411765-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:30.694614 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:30.694418 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85a4b13-e50d-40e5-91a4-64af9a411765-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:31.328980 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:31.328946 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c501563-7lfp4" event={"ID":"d85a4b13-e50d-40e5-91a4-64af9a411765","Type":"ContainerDied","Data":"144df70f8b130ecc5e563d3f0691af7f069811f7f91a0f063d28b37ede707cc3"} Oct 30 22:23:31.328980 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:31.328984 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c501563-7lfp4" Oct 30 22:23:31.329192 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:31.328992 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="144df70f8b130ecc5e563d3f0691af7f069811f7f91a0f063d28b37ede707cc3" Oct 30 22:23:49.180591 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:49.180556 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c501563-7lfp4"] Oct 30 22:23:49.226318 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:49.226284 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c501563-7lfp4"] Oct 30 22:23:49.683272 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:23:49.683231 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:23:50.573636 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:50.573600 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d85a4b13-e50d-40e5-91a4-64af9a411765" path="/var/lib/kubelet/pods/d85a4b13-e50d-40e5-91a4-64af9a411765/volumes" Oct 30 22:23:51.490454 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:23:51.490422 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:24:14.598243 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:24:14.598209 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:24:19.683658 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:24:19.683615 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:24:20.047865 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:24:20.047832 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c5118f8-9kzcp"] Oct 30 22:24:20.094327 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:24:20.094301 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c5118f8-9kzcp"] Oct 30 22:24:20.573069 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:24:20.573028 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c40445ab-df13-4409-b048-b5baea2586e9" path="/var/lib/kubelet/pods/c40445ab-df13-4409-b048-b5baea2586e9/volumes" Oct 30 22:24:49.683991 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:24:49.683944 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:25:19.683498 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:25:19.683414 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:25:45.575642 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:45.575606 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8123f1-njrjr"] Oct 30 22:25:45.576190 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:45.575913 4560 cpu_manager.go:414] "containerMap: removing container" podUID="d85a4b13-e50d-40e5-91a4-64af9a411765" containerName="revoke-keys" Oct 30 22:25:45.576190 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:45.575925 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d85a4b13-e50d-40e5-91a4-64af9a411765" containerName="revoke-keys" Oct 30 22:25:45.576190 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:45.575993 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="d85a4b13-e50d-40e5-91a4-64af9a411765" containerName="revoke-keys" Oct 30 22:25:45.578732 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:45.578711 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8123f1-njrjr" Oct 30 22:25:45.643685 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:45.643643 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8123f1-njrjr"] Oct 30 22:25:45.688078 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:45.688038 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:25:45.688078 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:45.688051 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:25:45.688078 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:45.688051 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:25:45.688327 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:45.688110 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7b9qb\"" Oct 30 22:25:45.690584 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:45.690562 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d1c3ed6-7eb4-496c-9444-8f3c91128554-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8123f1-njrjr\" (UID: \"5d1c3ed6-7eb4-496c-9444-8f3c91128554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8123f1-njrjr" Oct 30 22:25:45.690708 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:45.690601 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d1c3ed6-7eb4-496c-9444-8f3c91128554-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8123f1-njrjr\" (UID: \"5d1c3ed6-7eb4-496c-9444-8f3c91128554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8123f1-njrjr" Oct 30 22:25:45.690771 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:45.690734 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nj2j\" (UniqueName: \"kubernetes.io/projected/5d1c3ed6-7eb4-496c-9444-8f3c91128554-kube-api-access-7nj2j\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8123f1-njrjr\" (UID: \"5d1c3ed6-7eb4-496c-9444-8f3c91128554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8123f1-njrjr" Oct 30 22:25:45.791852 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:45.791813 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nj2j\" (UniqueName: \"kubernetes.io/projected/5d1c3ed6-7eb4-496c-9444-8f3c91128554-kube-api-access-7nj2j\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8123f1-njrjr\" (UID: \"5d1c3ed6-7eb4-496c-9444-8f3c91128554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8123f1-njrjr" Oct 30 22:25:45.792042 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:45.791874 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d1c3ed6-7eb4-496c-9444-8f3c91128554-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8123f1-njrjr\" (UID: \"5d1c3ed6-7eb4-496c-9444-8f3c91128554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8123f1-njrjr" Oct 30 22:25:45.792042 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:45.791894 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d1c3ed6-7eb4-496c-9444-8f3c91128554-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8123f1-njrjr\" (UID: \"5d1c3ed6-7eb4-496c-9444-8f3c91128554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8123f1-njrjr" Oct 30 22:25:45.792649 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:45.792629 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d1c3ed6-7eb4-496c-9444-8f3c91128554-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8123f1-njrjr\" (UID: \"5d1c3ed6-7eb4-496c-9444-8f3c91128554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8123f1-njrjr" Oct 30 22:25:45.794236 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:45.794217 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d1c3ed6-7eb4-496c-9444-8f3c91128554-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8123f1-njrjr\" (UID: \"5d1c3ed6-7eb4-496c-9444-8f3c91128554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8123f1-njrjr" Oct 30 22:25:45.849893 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:45.849819 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nj2j\" (UniqueName: \"kubernetes.io/projected/5d1c3ed6-7eb4-496c-9444-8f3c91128554-kube-api-access-7nj2j\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8123f1-njrjr\" (UID: \"5d1c3ed6-7eb4-496c-9444-8f3c91128554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8123f1-njrjr" Oct 30 22:25:45.890844 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:45.890816 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8123f1-njrjr" Oct 30 22:25:46.261064 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:46.261022 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8123f1-njrjr"] Oct 30 22:25:46.267974 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:25:46.267945 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d1c3ed6_7eb4_496c_9444_8f3c91128554.slice/crio-b4b83087d9da86346738de88ab4b7c7062d76e56944984bbb92f51f04a4be2f6 WatchSource:0}: Error finding container b4b83087d9da86346738de88ab4b7c7062d76e56944984bbb92f51f04a4be2f6: Status 404 returned error can't find the container with id b4b83087d9da86346738de88ab4b7c7062d76e56944984bbb92f51f04a4be2f6 Oct 30 22:25:46.894490 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:46.894456 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8123f1-njrjr" event={"ID":"5d1c3ed6-7eb4-496c-9444-8f3c91128554","Type":"ContainerStarted","Data":"4d24271390edcf97e47d063e872a8ac85ba0961ee20c666f87b9d77e6aafeac7"} Oct 30 22:25:46.894948 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:46.894513 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8123f1-njrjr" event={"ID":"5d1c3ed6-7eb4-496c-9444-8f3c91128554","Type":"ContainerStarted","Data":"b4b83087d9da86346738de88ab4b7c7062d76e56944984bbb92f51f04a4be2f6"} Oct 30 22:25:47.004774 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:47.004722 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8123f1-njrjr" podStartSLOduration=2.00470645 podStartE2EDuration="2.00470645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:47.003839241 +0000 UTC m=+4322.558539017" watchObservedRunningTime="2026-10-30 22:25:47.00470645 +0000 UTC m=+4322.559406225" Oct 30 22:25:47.899614 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:47.899578 4560 generic.go:356] "Generic (PLEG): container finished" podID="5d1c3ed6-7eb4-496c-9444-8f3c91128554" containerID="4d24271390edcf97e47d063e872a8ac85ba0961ee20c666f87b9d77e6aafeac7" exitCode=0 Oct 30 22:25:47.900026 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:47.899656 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8123f1-njrjr" event={"ID":"5d1c3ed6-7eb4-496c-9444-8f3c91128554","Type":"ContainerDied","Data":"4d24271390edcf97e47d063e872a8ac85ba0961ee20c666f87b9d77e6aafeac7"} Oct 30 22:25:48.192412 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:48.192304 4560 scope.go:122] "RemoveContainer" containerID="600f51599bd098bd5861220d7161d1b1e640b597d648d502c2750fb55a93d134" Oct 30 22:25:48.204813 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:48.204788 4560 scope.go:122] "RemoveContainer" containerID="8b01b6961d6d5ca87a41fdf7ccc672bce3ee9fb10c263e1a821a2c64212d4391" Oct 30 22:25:49.025905 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:49.025877 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8123f1-njrjr" Oct 30 22:25:49.120200 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:49.120164 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d1c3ed6-7eb4-496c-9444-8f3c91128554-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d1c3ed6-7eb4-496c-9444-8f3c91128554-maas-api-xks-ca\") pod \"5d1c3ed6-7eb4-496c-9444-8f3c91128554\" (UID: \"5d1c3ed6-7eb4-496c-9444-8f3c91128554\") " Oct 30 22:25:49.120396 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:49.120219 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d1c3ed6-7eb4-496c-9444-8f3c91128554-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d1c3ed6-7eb4-496c-9444-8f3c91128554-maas-api-openshift-ca\") pod \"5d1c3ed6-7eb4-496c-9444-8f3c91128554\" (UID: \"5d1c3ed6-7eb4-496c-9444-8f3c91128554\") " Oct 30 22:25:49.120396 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:49.120288 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d1c3ed6-7eb4-496c-9444-8f3c91128554-kube-api-access-7nj2j\" (UniqueName: \"kubernetes.io/projected/5d1c3ed6-7eb4-496c-9444-8f3c91128554-kube-api-access-7nj2j\") pod \"5d1c3ed6-7eb4-496c-9444-8f3c91128554\" (UID: \"5d1c3ed6-7eb4-496c-9444-8f3c91128554\") " Oct 30 22:25:49.120623 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:49.120602 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d1c3ed6-7eb4-496c-9444-8f3c91128554-maas-api-openshift-ca" pod "5d1c3ed6-7eb4-496c-9444-8f3c91128554" (UID: "5d1c3ed6-7eb4-496c-9444-8f3c91128554"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:49.122441 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:49.122416 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d1c3ed6-7eb4-496c-9444-8f3c91128554-kube-api-access-7nj2j" pod "5d1c3ed6-7eb4-496c-9444-8f3c91128554" (UID: "5d1c3ed6-7eb4-496c-9444-8f3c91128554"). InnerVolumeSpecName "kube-api-access-7nj2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:49.122441 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:49.122433 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d1c3ed6-7eb4-496c-9444-8f3c91128554-maas-api-xks-ca" pod "5d1c3ed6-7eb4-496c-9444-8f3c91128554" (UID: "5d1c3ed6-7eb4-496c-9444-8f3c91128554"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:25:49.221770 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:49.221672 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d1c3ed6-7eb4-496c-9444-8f3c91128554-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:49.221770 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:49.221718 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d1c3ed6-7eb4-496c-9444-8f3c91128554-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:49.221770 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:49.221732 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nj2j\" (UniqueName: \"kubernetes.io/projected/5d1c3ed6-7eb4-496c-9444-8f3c91128554-kube-api-access-7nj2j\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:49.684086 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:25:49.684050 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:25:49.908313 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:49.908282 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8123f1-njrjr" Oct 30 22:25:49.908510 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:49.908284 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8123f1-njrjr" event={"ID":"5d1c3ed6-7eb4-496c-9444-8f3c91128554","Type":"ContainerDied","Data":"b4b83087d9da86346738de88ab4b7c7062d76e56944984bbb92f51f04a4be2f6"} Oct 30 22:25:49.908510 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:25:49.908414 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4b83087d9da86346738de88ab4b7c7062d76e56944984bbb92f51f04a4be2f6" Oct 30 22:26:08.713195 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:08.713160 4560 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8123f1-njrjr"] Oct 30 22:26:08.760025 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:08.759984 4560 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8123f1-njrjr"] Oct 30 22:26:10.574348 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:10.574315 4560 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d1c3ed6-7eb4-496c-9444-8f3c91128554" path="/var/lib/kubelet/pods/5d1c3ed6-7eb4-496c-9444-8f3c91128554/volumes" Oct 30 22:26:10.989387 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:10.989355 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:26:19.683803 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:26:19.683769 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:26:24.901032 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:24.900990 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:26:35.585623 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:35.585536 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:26:43.784965 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:43.784924 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tbdjj"] Oct 30 22:26:43.785495 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:43.785383 4560 cpu_manager.go:414] "containerMap: removing container" podUID="5d1c3ed6-7eb4-496c-9444-8f3c91128554" containerName="revoke-keys" Oct 30 22:26:43.785495 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:43.785403 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d1c3ed6-7eb4-496c-9444-8f3c91128554" containerName="revoke-keys" Oct 30 22:26:43.785626 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:43.785507 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d1c3ed6-7eb4-496c-9444-8f3c91128554" containerName="revoke-keys" Oct 30 22:26:43.788447 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:43.788426 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tbdjj" Oct 30 22:26:43.836423 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:43.836394 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-tbdjj\" (UID: \"fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tbdjj" Oct 30 22:26:43.836569 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:43.836428 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxlk5\" (UniqueName: \"kubernetes.io/projected/fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9-kube-api-access-jxlk5\") pod \"maas-api-revoke-sub-f03629db-tbdjj\" (UID: \"fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tbdjj" Oct 30 22:26:43.836650 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:43.836582 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-tbdjj\" (UID: \"fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tbdjj" Oct 30 22:26:43.838187 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:43.838168 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:26:43.885027 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:43.885000 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7b9qb\"" Oct 30 22:26:43.885027 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:43.885011 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:26:43.885179 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:43.885093 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:26:43.886359 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:43.886329 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tbdjj"] Oct 30 22:26:43.937226 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:43.937200 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-tbdjj\" (UID: \"fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tbdjj" Oct 30 22:26:43.937369 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:43.937270 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-tbdjj\" (UID: \"fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tbdjj" Oct 30 22:26:43.937369 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:43.937291 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxlk5\" (UniqueName: \"kubernetes.io/projected/fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9-kube-api-access-jxlk5\") pod \"maas-api-revoke-sub-f03629db-tbdjj\" (UID: \"fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tbdjj" Oct 30 22:26:43.937871 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:43.937845 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-tbdjj\" (UID: \"fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tbdjj" Oct 30 22:26:43.939493 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:43.939470 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-tbdjj\" (UID: \"fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tbdjj" Oct 30 22:26:43.996642 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:43.996618 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxlk5\" (UniqueName: \"kubernetes.io/projected/fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9-kube-api-access-jxlk5\") pod \"maas-api-revoke-sub-f03629db-tbdjj\" (UID: \"fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tbdjj" Oct 30 22:26:44.100378 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:44.100291 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tbdjj" Oct 30 22:26:44.473247 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:44.473172 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tbdjj"] Oct 30 22:26:44.479782 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:26:44.479752 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc98b89_7cfe_4dfd_8edc_34e1bfab6ea9.slice/crio-d0ef95575cb4ff5bf1359be0588c182ac5d6e73c50bf1b79ccd73a695a076574 WatchSource:0}: Error finding container d0ef95575cb4ff5bf1359be0588c182ac5d6e73c50bf1b79ccd73a695a076574: Status 404 returned error can't find the container with id d0ef95575cb4ff5bf1359be0588c182ac5d6e73c50bf1b79ccd73a695a076574 Oct 30 22:26:45.145772 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:45.145732 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tbdjj" event={"ID":"fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9","Type":"ContainerStarted","Data":"aec9f27953346f18301a33641a1e156e06f2bb5d539bb2c122ebf6ef7077808f"} Oct 30 22:26:45.146275 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:45.145792 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tbdjj" event={"ID":"fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9","Type":"ContainerStarted","Data":"d0ef95575cb4ff5bf1359be0588c182ac5d6e73c50bf1b79ccd73a695a076574"} Oct 30 22:26:45.256747 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:45.256688 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tbdjj" podStartSLOduration=2.256672633 podStartE2EDuration="2.256672633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:45.255607148 +0000 UTC m=+4380.810306927" watchObservedRunningTime="2026-10-30 22:26:45.256672633 +0000 UTC m=+4380.811372410" Oct 30 22:26:46.151437 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:46.151400 4560 generic.go:356] "Generic (PLEG): container finished" podID="fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9" containerID="aec9f27953346f18301a33641a1e156e06f2bb5d539bb2c122ebf6ef7077808f" exitCode=0 Oct 30 22:26:46.151853 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:46.151483 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tbdjj" event={"ID":"fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9","Type":"ContainerDied","Data":"aec9f27953346f18301a33641a1e156e06f2bb5d539bb2c122ebf6ef7077808f"} Oct 30 22:26:47.284405 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:47.284380 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tbdjj" Oct 30 22:26:47.368534 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:47.368504 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9-kube-api-access-jxlk5\" (UniqueName: \"kubernetes.io/projected/fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9-kube-api-access-jxlk5\") pod \"fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9\" (UID: \"fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9\") " Oct 30 22:26:47.368661 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:47.368540 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9-maas-api-xks-ca\") pod \"fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9\" (UID: \"fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9\") " Oct 30 22:26:47.368661 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:47.368632 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9-maas-api-openshift-ca\") pod \"fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9\" (UID: \"fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9\") " Oct 30 22:26:47.368973 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:47.368949 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9-maas-api-openshift-ca" pod "fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9" (UID: "fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:47.370490 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:47.370470 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9-kube-api-access-jxlk5" pod "fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9" (UID: "fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9"). InnerVolumeSpecName "kube-api-access-jxlk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:47.370591 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:47.370571 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9-maas-api-xks-ca" pod "fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9" (UID: "fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:26:47.469908 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:47.469834 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:47.469908 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:47.469864 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxlk5\" (UniqueName: \"kubernetes.io/projected/fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9-kube-api-access-jxlk5\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:47.469908 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:47.469879 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:48.161359 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:48.161312 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tbdjj" Oct 30 22:26:48.161538 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:48.161356 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tbdjj" event={"ID":"fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9","Type":"ContainerDied","Data":"d0ef95575cb4ff5bf1359be0588c182ac5d6e73c50bf1b79ccd73a695a076574"} Oct 30 22:26:48.161538 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:48.161405 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0ef95575cb4ff5bf1359be0588c182ac5d6e73c50bf1b79ccd73a695a076574" Oct 30 22:26:49.684136 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:26:49.684091 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:26:56.765717 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:26:56.765685 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:27:03.102699 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:03.102658 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-smhb8"] Oct 30 22:27:03.103246 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:03.103144 4560 cpu_manager.go:414] "containerMap: removing container" podUID="fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9" containerName="revoke-keys" Oct 30 22:27:03.103246 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:03.103163 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9" containerName="revoke-keys" Oct 30 22:27:03.103427 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:03.103247 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9" containerName="revoke-keys" Oct 30 22:27:03.106711 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:03.106684 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-smhb8" Oct 30 22:27:03.167364 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:03.163806 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:27:03.167364 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:03.163972 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:27:03.167364 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:03.164005 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:27:03.209687 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:03.209650 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7b9qb\"" Oct 30 22:27:03.210862 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:03.210830 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea18004c-ef6d-4343-9514-16459bebbc09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-smhb8\" (UID: \"ea18004c-ef6d-4343-9514-16459bebbc09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-smhb8" Oct 30 22:27:03.211030 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:03.210883 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea18004c-ef6d-4343-9514-16459bebbc09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-smhb8\" (UID: \"ea18004c-ef6d-4343-9514-16459bebbc09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-smhb8" Oct 30 22:27:03.211030 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:03.210981 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqdvs\" (UniqueName: \"kubernetes.io/projected/ea18004c-ef6d-4343-9514-16459bebbc09-kube-api-access-jqdvs\") pod \"maas-api-revoke-sub-0165d7d2-smhb8\" (UID: \"ea18004c-ef6d-4343-9514-16459bebbc09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-smhb8" Oct 30 22:27:03.212680 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:03.212652 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-smhb8"] Oct 30 22:27:03.311593 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:03.311548 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqdvs\" (UniqueName: \"kubernetes.io/projected/ea18004c-ef6d-4343-9514-16459bebbc09-kube-api-access-jqdvs\") pod \"maas-api-revoke-sub-0165d7d2-smhb8\" (UID: \"ea18004c-ef6d-4343-9514-16459bebbc09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-smhb8" Oct 30 22:27:03.311933 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:03.311910 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea18004c-ef6d-4343-9514-16459bebbc09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-smhb8\" (UID: \"ea18004c-ef6d-4343-9514-16459bebbc09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-smhb8" Oct 30 22:27:03.311988 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:03.311950 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea18004c-ef6d-4343-9514-16459bebbc09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-smhb8\" (UID: \"ea18004c-ef6d-4343-9514-16459bebbc09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-smhb8" Oct 30 22:27:03.312689 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:03.312667 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea18004c-ef6d-4343-9514-16459bebbc09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-smhb8\" (UID: \"ea18004c-ef6d-4343-9514-16459bebbc09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-smhb8" Oct 30 22:27:03.314695 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:03.314678 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea18004c-ef6d-4343-9514-16459bebbc09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-smhb8\" (UID: \"ea18004c-ef6d-4343-9514-16459bebbc09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-smhb8" Oct 30 22:27:03.399665 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:03.399587 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqdvs\" (UniqueName: \"kubernetes.io/projected/ea18004c-ef6d-4343-9514-16459bebbc09-kube-api-access-jqdvs\") pod \"maas-api-revoke-sub-0165d7d2-smhb8\" (UID: \"ea18004c-ef6d-4343-9514-16459bebbc09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-smhb8" Oct 30 22:27:03.418387 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:03.418361 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-smhb8" Oct 30 22:27:03.932843 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:03.932764 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-smhb8"] Oct 30 22:27:04.233823 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:04.233782 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-smhb8" event={"ID":"ea18004c-ef6d-4343-9514-16459bebbc09","Type":"ContainerStarted","Data":"14758e0e949afa7a9aa6e2ead8f73c879eaafd61e0ebbbf875ff7038876d98a2"} Oct 30 22:27:05.239503 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:05.239469 4560 generic.go:356] "Generic (PLEG): container finished" podID="ea18004c-ef6d-4343-9514-16459bebbc09" containerID="4a4e21f288ebeb447116021f0253d340ab23350c580ba52f8860a0284429263f" exitCode=0 Oct 30 22:27:05.239899 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:05.239541 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-smhb8" event={"ID":"ea18004c-ef6d-4343-9514-16459bebbc09","Type":"ContainerDied","Data":"4a4e21f288ebeb447116021f0253d340ab23350c580ba52f8860a0284429263f"} Oct 30 22:27:06.368724 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:06.368699 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-smhb8" Oct 30 22:27:06.437522 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:06.437494 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea18004c-ef6d-4343-9514-16459bebbc09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea18004c-ef6d-4343-9514-16459bebbc09-maas-api-openshift-ca\") pod \"ea18004c-ef6d-4343-9514-16459bebbc09\" (UID: \"ea18004c-ef6d-4343-9514-16459bebbc09\") " Oct 30 22:27:06.437664 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:06.437577 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea18004c-ef6d-4343-9514-16459bebbc09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea18004c-ef6d-4343-9514-16459bebbc09-maas-api-xks-ca\") pod \"ea18004c-ef6d-4343-9514-16459bebbc09\" (UID: \"ea18004c-ef6d-4343-9514-16459bebbc09\") " Oct 30 22:27:06.437664 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:06.437623 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea18004c-ef6d-4343-9514-16459bebbc09-kube-api-access-jqdvs\" (UniqueName: \"kubernetes.io/projected/ea18004c-ef6d-4343-9514-16459bebbc09-kube-api-access-jqdvs\") pod \"ea18004c-ef6d-4343-9514-16459bebbc09\" (UID: \"ea18004c-ef6d-4343-9514-16459bebbc09\") " Oct 30 22:27:06.437878 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:06.437852 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea18004c-ef6d-4343-9514-16459bebbc09-maas-api-openshift-ca" pod "ea18004c-ef6d-4343-9514-16459bebbc09" (UID: "ea18004c-ef6d-4343-9514-16459bebbc09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:27:06.439549 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:06.439519 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea18004c-ef6d-4343-9514-16459bebbc09-maas-api-xks-ca" pod "ea18004c-ef6d-4343-9514-16459bebbc09" (UID: "ea18004c-ef6d-4343-9514-16459bebbc09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:27:06.439655 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:06.439594 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea18004c-ef6d-4343-9514-16459bebbc09-kube-api-access-jqdvs" pod "ea18004c-ef6d-4343-9514-16459bebbc09" (UID: "ea18004c-ef6d-4343-9514-16459bebbc09"). InnerVolumeSpecName "kube-api-access-jqdvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:06.538275 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:06.538203 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea18004c-ef6d-4343-9514-16459bebbc09-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:06.538275 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:06.538228 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea18004c-ef6d-4343-9514-16459bebbc09-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:06.538275 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:06.538239 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqdvs\" (UniqueName: \"kubernetes.io/projected/ea18004c-ef6d-4343-9514-16459bebbc09-kube-api-access-jqdvs\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:07.249002 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:07.248963 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-smhb8" event={"ID":"ea18004c-ef6d-4343-9514-16459bebbc09","Type":"ContainerDied","Data":"14758e0e949afa7a9aa6e2ead8f73c879eaafd61e0ebbbf875ff7038876d98a2"} Oct 30 22:27:07.249002 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:07.248995 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-smhb8" Oct 30 22:27:07.249220 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:07.249022 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14758e0e949afa7a9aa6e2ead8f73c879eaafd61e0ebbbf875ff7038876d98a2" Oct 30 22:27:14.974083 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:14.974045 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ghpmt"] Oct 30 22:27:14.974506 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:14.974393 4560 cpu_manager.go:414] "containerMap: removing container" podUID="ea18004c-ef6d-4343-9514-16459bebbc09" containerName="revoke-keys" Oct 30 22:27:14.974506 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:14.974409 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea18004c-ef6d-4343-9514-16459bebbc09" containerName="revoke-keys" Oct 30 22:27:14.974506 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:14.974491 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea18004c-ef6d-4343-9514-16459bebbc09" containerName="revoke-keys" Oct 30 22:27:14.977440 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:14.977417 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ghpmt" Oct 30 22:27:15.029212 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:15.029186 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:27:15.075893 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:15.075866 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:27:15.076048 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:15.075906 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:27:15.076048 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:15.075962 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7b9qb\"" Oct 30 22:27:15.077390 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:15.077372 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ghpmt"] Oct 30 22:27:15.107720 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:15.107689 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0877b681-99ed-486f-b61f-f3abb7e78651-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-ghpmt\" (UID: \"0877b681-99ed-486f-b61f-f3abb7e78651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ghpmt" Oct 30 22:27:15.107844 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:15.107733 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwj9l\" (UniqueName: \"kubernetes.io/projected/0877b681-99ed-486f-b61f-f3abb7e78651-kube-api-access-lwj9l\") pod \"maas-api-revoke-sub-6b19338f-ghpmt\" (UID: \"0877b681-99ed-486f-b61f-f3abb7e78651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ghpmt" Oct 30 22:27:15.107894 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:15.107858 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0877b681-99ed-486f-b61f-f3abb7e78651-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-ghpmt\" (UID: \"0877b681-99ed-486f-b61f-f3abb7e78651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ghpmt" Oct 30 22:27:15.208527 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:15.208497 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0877b681-99ed-486f-b61f-f3abb7e78651-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-ghpmt\" (UID: \"0877b681-99ed-486f-b61f-f3abb7e78651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ghpmt" Oct 30 22:27:15.208700 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:15.208539 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwj9l\" (UniqueName: \"kubernetes.io/projected/0877b681-99ed-486f-b61f-f3abb7e78651-kube-api-access-lwj9l\") pod \"maas-api-revoke-sub-6b19338f-ghpmt\" (UID: \"0877b681-99ed-486f-b61f-f3abb7e78651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ghpmt" Oct 30 22:27:15.208700 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:15.208598 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0877b681-99ed-486f-b61f-f3abb7e78651-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-ghpmt\" (UID: \"0877b681-99ed-486f-b61f-f3abb7e78651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ghpmt" Oct 30 22:27:15.209298 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:15.209278 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0877b681-99ed-486f-b61f-f3abb7e78651-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-ghpmt\" (UID: \"0877b681-99ed-486f-b61f-f3abb7e78651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ghpmt" Oct 30 22:27:15.210970 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:15.210951 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0877b681-99ed-486f-b61f-f3abb7e78651-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-ghpmt\" (UID: \"0877b681-99ed-486f-b61f-f3abb7e78651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ghpmt" Oct 30 22:27:15.266662 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:15.266638 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwj9l\" (UniqueName: \"kubernetes.io/projected/0877b681-99ed-486f-b61f-f3abb7e78651-kube-api-access-lwj9l\") pod \"maas-api-revoke-sub-6b19338f-ghpmt\" (UID: \"0877b681-99ed-486f-b61f-f3abb7e78651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ghpmt" Oct 30 22:27:15.289740 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:15.289717 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ghpmt" Oct 30 22:27:15.655631 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:15.655558 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ghpmt"] Oct 30 22:27:15.662382 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:27:15.662353 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0877b681_99ed_486f_b61f_f3abb7e78651.slice/crio-9a5f029bda9c76a406b6019bd4ea1d56670badc5ac3439c88c4affd3e2a3e1db WatchSource:0}: Error finding container 9a5f029bda9c76a406b6019bd4ea1d56670badc5ac3439c88c4affd3e2a3e1db: Status 404 returned error can't find the container with id 9a5f029bda9c76a406b6019bd4ea1d56670badc5ac3439c88c4affd3e2a3e1db Oct 30 22:27:16.287303 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:16.287269 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ghpmt" event={"ID":"0877b681-99ed-486f-b61f-f3abb7e78651","Type":"ContainerStarted","Data":"5239d2f03601286810083278ce8250862ff3afc32a466fdb9a8117dc9a24a2e2"} Oct 30 22:27:16.287730 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:16.287373 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ghpmt" event={"ID":"0877b681-99ed-486f-b61f-f3abb7e78651","Type":"ContainerStarted","Data":"9a5f029bda9c76a406b6019bd4ea1d56670badc5ac3439c88c4affd3e2a3e1db"} Oct 30 22:27:16.397414 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:16.397358 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ghpmt" podStartSLOduration=2.397332631 podStartE2EDuration="2.397332631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:16.3968424 +0000 UTC m=+4411.951542178" watchObservedRunningTime="2026-10-30 22:27:16.397332631 +0000 UTC m=+4411.952032410" Oct 30 22:27:17.292430 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:17.292400 4560 generic.go:356] "Generic (PLEG): container finished" podID="0877b681-99ed-486f-b61f-f3abb7e78651" containerID="5239d2f03601286810083278ce8250862ff3afc32a466fdb9a8117dc9a24a2e2" exitCode=0 Oct 30 22:27:17.294735 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:17.292492 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ghpmt" event={"ID":"0877b681-99ed-486f-b61f-f3abb7e78651","Type":"ContainerDied","Data":"5239d2f03601286810083278ce8250862ff3afc32a466fdb9a8117dc9a24a2e2"} Oct 30 22:27:18.420379 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:18.420354 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ghpmt" Oct 30 22:27:18.543052 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:18.543020 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0877b681-99ed-486f-b61f-f3abb7e78651-kube-api-access-lwj9l\" (UniqueName: \"kubernetes.io/projected/0877b681-99ed-486f-b61f-f3abb7e78651-kube-api-access-lwj9l\") pod \"0877b681-99ed-486f-b61f-f3abb7e78651\" (UID: \"0877b681-99ed-486f-b61f-f3abb7e78651\") " Oct 30 22:27:18.543195 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:18.543071 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0877b681-99ed-486f-b61f-f3abb7e78651-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0877b681-99ed-486f-b61f-f3abb7e78651-maas-api-openshift-ca\") pod \"0877b681-99ed-486f-b61f-f3abb7e78651\" (UID: \"0877b681-99ed-486f-b61f-f3abb7e78651\") " Oct 30 22:27:18.543195 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:18.543120 4560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0877b681-99ed-486f-b61f-f3abb7e78651-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0877b681-99ed-486f-b61f-f3abb7e78651-maas-api-xks-ca\") pod \"0877b681-99ed-486f-b61f-f3abb7e78651\" (UID: \"0877b681-99ed-486f-b61f-f3abb7e78651\") " Oct 30 22:27:18.543478 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:18.543453 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0877b681-99ed-486f-b61f-f3abb7e78651-maas-api-openshift-ca" pod "0877b681-99ed-486f-b61f-f3abb7e78651" (UID: "0877b681-99ed-486f-b61f-f3abb7e78651"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:27:18.545196 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:18.545167 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0877b681-99ed-486f-b61f-f3abb7e78651-maas-api-xks-ca" pod "0877b681-99ed-486f-b61f-f3abb7e78651" (UID: "0877b681-99ed-486f-b61f-f3abb7e78651"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:27:18.545311 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:18.545231 4560 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0877b681-99ed-486f-b61f-f3abb7e78651-kube-api-access-lwj9l" pod "0877b681-99ed-486f-b61f-f3abb7e78651" (UID: "0877b681-99ed-486f-b61f-f3abb7e78651"). InnerVolumeSpecName "kube-api-access-lwj9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:18.644572 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:18.644546 4560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwj9l\" (UniqueName: \"kubernetes.io/projected/0877b681-99ed-486f-b61f-f3abb7e78651-kube-api-access-lwj9l\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:18.644572 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:18.644570 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0877b681-99ed-486f-b61f-f3abb7e78651-maas-api-openshift-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:18.644709 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:18.644580 4560 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0877b681-99ed-486f-b61f-f3abb7e78651-maas-api-xks-ca\") on node \"ip-10-0-129-83.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:19.301274 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:19.301246 4560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ghpmt" Oct 30 22:27:19.301468 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:19.301240 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ghpmt" event={"ID":"0877b681-99ed-486f-b61f-f3abb7e78651","Type":"ContainerDied","Data":"9a5f029bda9c76a406b6019bd4ea1d56670badc5ac3439c88c4affd3e2a3e1db"} Oct 30 22:27:19.301468 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:19.301387 4560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a5f029bda9c76a406b6019bd4ea1d56670badc5ac3439c88c4affd3e2a3e1db" Oct 30 22:27:19.683784 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:27:19.683706 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:27:27.880568 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:27.880533 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-475wg"] Oct 30 22:27:39.716351 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:39.716292 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-5gk2d_12fcc260-c72a-44da-a01a-f654ccbe1c6e/manager/0.log" Oct 30 22:27:40.635165 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:40.635138 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-ljvd8_9bae0774-2ade-4011-a36e-e0c36d5beee2/server/0.log" Oct 30 22:27:42.449136 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:42.449109 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-smhb8_ea18004c-ef6d-4343-9514-16459bebbc09/revoke-keys/0.log" Oct 30 22:27:42.670011 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:42.669985 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-ghpmt_0877b681-99ed-486f-b61f-f3abb7e78651/revoke-keys/0.log" Oct 30 22:27:42.898410 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:42.898379 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-tbdjj_fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9/revoke-keys/0.log" Oct 30 22:27:45.585215 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:45.585178 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-ft7rc_ba6cc181-9632-4861-ab91-0e0836e16f49/kuadrant-console-plugin/0.log" Oct 30 22:27:46.198449 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:46.198418 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-475wg_b4ea7858-6e8b-4fa8-a805-0d82678b5dac/limitador/0.log" Oct 30 22:27:47.523004 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:47.522976 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-ngwjn_d9bb2bf8-3fce-443b-a561-ed34cc9e9c61/discovery/0.log" Oct 30 22:27:48.112521 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:48.112482 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6645f5c86b-xmshw_b7f14427-2d3d-4a1d-a919-2842cded93b0/kube-auth-proxy/0.log" Oct 30 22:27:48.359918 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:48.359886 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-gl8p4_4c21f641-8dd8-4549-8748-94c9fdbd8ba2/istio-proxy/0.log" Oct 30 22:27:49.365101 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:49.365071 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-m2sxq_c73ef128-5710-4654-a155-4b9083ab04bb/istio-proxy/0.log" Oct 30 22:27:49.683737 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:27:49.683650 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:27:50.574149 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:27:50.574112 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-zfpbl_2e6e5570-8209-4ff2-84d2-1bf88ca77c81/istio-proxy/0.log" Oct 30 22:28:05.802126 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:28:05.802054 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-5gk2d_12fcc260-c72a-44da-a01a-f654ccbe1c6e/manager/0.log" Oct 30 22:28:06.640270 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:28:06.640245 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-ljvd8_9bae0774-2ade-4011-a36e-e0c36d5beee2/server/0.log" Oct 30 22:28:08.401027 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:28:08.400996 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-smhb8_ea18004c-ef6d-4343-9514-16459bebbc09/revoke-keys/0.log" Oct 30 22:28:08.646049 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:28:08.646020 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-ghpmt_0877b681-99ed-486f-b61f-f3abb7e78651/revoke-keys/0.log" Oct 30 22:28:08.897502 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:28:08.897472 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-tbdjj_fcc98b89-7cfe-4dfd-8edc-34e1bfab6ea9/revoke-keys/0.log" Oct 30 22:28:11.453414 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:28:11.453379 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-ft7rc_ba6cc181-9632-4861-ab91-0e0836e16f49/kuadrant-console-plugin/0.log" Oct 30 22:28:12.114442 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:28:12.114412 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-475wg_b4ea7858-6e8b-4fa8-a805-0d82678b5dac/limitador/0.log" Oct 30 22:28:13.325954 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:28:13.325922 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-ngwjn_d9bb2bf8-3fce-443b-a561-ed34cc9e9c61/discovery/0.log" Oct 30 22:28:13.973633 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:28:13.973602 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6645f5c86b-xmshw_b7f14427-2d3d-4a1d-a919-2842cded93b0/kube-auth-proxy/0.log" Oct 30 22:28:14.231655 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:28:14.231628 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-gl8p4_4c21f641-8dd8-4549-8748-94c9fdbd8ba2/istio-proxy/0.log" Oct 30 22:28:15.473946 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:28:15.473916 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-m2sxq_c73ef128-5710-4654-a155-4b9083ab04bb/istio-proxy/0.log" Oct 30 22:28:16.943265 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:28:16.943234 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-zfpbl_2e6e5570-8209-4ff2-84d2-1bf88ca77c81/istio-proxy/0.log" Oct 30 22:28:19.683782 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:28:19.683746 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:28:49.684114 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:28:49.684070 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:29:19.684103 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:29:19.684041 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:29:44.905931 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:29:44.905901 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jztdg_3341bf47-8c8f-4073-8758-122e04e0b138/global-pull-secret-syncer/0.log" Oct 30 22:29:47.704063 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:29:47.704037 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vpxpj_08f1ac32-f49e-4aa2-9ab8-8a1aac7bef36/konnectivity-agent/0.log" Oct 30 22:29:48.241535 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:29:48.241503 4560 scope.go:122] "RemoveContainer" containerID="b7f45670f7488509abe1a322e97052ccefb7382299d0380f620dd60c5cbf459d" Oct 30 22:29:48.311212 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:29:48.311183 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-83.us-west-2.compute.internal_8b9addd42169977c49b17223d2454db0/haproxy/0.log" Oct 30 22:29:49.683238 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:29:49.683203 4560 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hq4sb" containerName="extractor" Oct 30 22:29:52.334718 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:29:52.334692 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-ft7rc_ba6cc181-9632-4861-ab91-0e0836e16f49/kuadrant-console-plugin/0.log" Oct 30 22:29:52.698973 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:29:52.698896 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hgcw8/perf-node-gather-daemonset-x6ll8"] Oct 30 22:29:52.699229 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:29:52.699216 4560 cpu_manager.go:414] "containerMap: removing container" podUID="0877b681-99ed-486f-b61f-f3abb7e78651" containerName="revoke-keys" Oct 30 22:29:52.699276 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:29:52.699230 4560 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0877b681-99ed-486f-b61f-f3abb7e78651" containerName="revoke-keys" Oct 30 22:29:52.699313 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:29:52.699287 4560 memory_manager.go:365] "RemoveStaleState removing state" podUID="0877b681-99ed-486f-b61f-f3abb7e78651" containerName="revoke-keys" Oct 30 22:29:52.703279 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:29:52.703260 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-x6ll8" Oct 30 22:29:52.754381 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:29:52.754356 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hgcw8\"/\"kube-root-ca.crt\"" Oct 30 22:29:52.754534 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:29:52.754442 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hgcw8\"/\"openshift-service-ca.crt\"" Oct 30 22:29:52.798565 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:29:52.798537 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d73912b-221e-4ff0-8144-af701cb79918-sys\") pod \"perf-node-gather-daemonset-x6ll8\" (UID: \"2d73912b-221e-4ff0-8144-af701cb79918\") " pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-x6ll8" Oct 30 22:29:52.798694 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:29:52.798574 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njjcr\" (UniqueName: \"kubernetes.io/projected/2d73912b-221e-4ff0-8144-af701cb79918-kube-api-access-njjcr\") pod \"perf-node-gather-daemonset-x6ll8\" (UID: \"2d73912b-221e-4ff0-8144-af701cb79918\") " pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-x6ll8" Oct 30 22:29:52.798694 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:29:52.798600 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2d73912b-221e-4ff0-8144-af701cb79918-lib-modules\") pod \"perf-node-gather-daemonset-x6ll8\" (UID: \"2d73912b-221e-4ff0-8144-af701cb79918\") " pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-x6ll8" Oct 30 22:29:52.798792 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:29:52.798726 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2d73912b-221e-4ff0-8144-af701cb79918-podres\") pod \"perf-node-gather-daemonset-x6ll8\" (UID: \"2d73912b-221e-4ff0-8144-af701cb79918\") " pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-x6ll8" Oct 30 22:29:52.798792 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:29:52.798752 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2d73912b-221e-4ff0-8144-af701cb79918-proc\") pod \"perf-node-gather-daemonset-x6ll8\" (UID: \"2d73912b-221e-4ff0-8144-af701cb79918\") " pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-x6ll8" Oct 30 22:29:52.800610 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:29:52.800590 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hgcw8\"/\"default-dockercfg-sqfbc\"" Oct 30 22:29:52.801899 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:29:52.801878 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hgcw8/perf-node-gather-daemonset-x6ll8"] Oct 30 22:29:52.899743 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:29:52.899711 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2d73912b-221e-4ff0-8144-af701cb79918-lib-modules\") pod \"perf-node-gather-daemonset-x6ll8\" (UID: \"2d73912b-221e-4ff0-8144-af701cb79918\") " pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-x6ll8" Oct 30 22:29:52.899934 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:29:52.899750 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2d73912b-221e-4ff0-8144-af701cb79918-podres\") pod \"perf-node-gather-daemonset-x6ll8\" (UID: \"2d73912b-221e-4ff0-8144-af701cb79918\") " pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-x6ll8" Oct 30 22:29:52.899934 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:29:52.899777 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2d73912b-221e-4ff0-8144-af701cb79918-proc\") pod \"perf-node-gather-daemonset-x6ll8\" (UID: \"2d73912b-221e-4ff0-8144-af701cb79918\") " pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-x6ll8" Oct 30 22:29:52.899934 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:29:52.899856 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2d73912b-221e-4ff0-8144-af701cb79918-lib-modules\") pod \"perf-node-gather-daemonset-x6ll8\" (UID: \"2d73912b-221e-4ff0-8144-af701cb79918\") " pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-x6ll8" Oct 30 22:29:52.899934 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:29:52.899869 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d73912b-221e-4ff0-8144-af701cb79918-sys\") pod \"perf-node-gather-daemonset-x6ll8\" (UID: \"2d73912b-221e-4ff0-8144-af701cb79918\") " pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-x6ll8" Oct 30 22:29:52.899934 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:29:52.899867 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2d73912b-221e-4ff0-8144-af701cb79918-podres\") pod \"perf-node-gather-daemonset-x6ll8\" (UID: \"2d73912b-221e-4ff0-8144-af701cb79918\") " pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-x6ll8" Oct 30 22:29:52.899934 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:29:52.899915 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njjcr\" (UniqueName: \"kubernetes.io/projected/2d73912b-221e-4ff0-8144-af701cb79918-kube-api-access-njjcr\") pod \"perf-node-gather-daemonset-x6ll8\" (UID: \"2d73912b-221e-4ff0-8144-af701cb79918\") " pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-x6ll8" Oct 30 22:29:52.899934 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:29:52.899919 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2d73912b-221e-4ff0-8144-af701cb79918-proc\") pod \"perf-node-gather-daemonset-x6ll8\" (UID: \"2d73912b-221e-4ff0-8144-af701cb79918\") " pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-x6ll8" Oct 30 22:29:52.899934 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:29:52.899933 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d73912b-221e-4ff0-8144-af701cb79918-sys\") pod \"perf-node-gather-daemonset-x6ll8\" (UID: \"2d73912b-221e-4ff0-8144-af701cb79918\") " pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-x6ll8" Oct 30 22:29:52.957666 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:29:52.957606 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njjcr\" (UniqueName: \"kubernetes.io/projected/2d73912b-221e-4ff0-8144-af701cb79918-kube-api-access-njjcr\") pod \"perf-node-gather-daemonset-x6ll8\" (UID: \"2d73912b-221e-4ff0-8144-af701cb79918\") " pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-x6ll8" Oct 30 22:29:53.015606 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:29:53.015579 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-x6ll8" Oct 30 22:29:53.157796 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:29:53.157766 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-475wg_b4ea7858-6e8b-4fa8-a805-0d82678b5dac/limitador/0.log" Oct 30 22:29:53.383603 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:29:53.383569 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hgcw8/perf-node-gather-daemonset-x6ll8"] Oct 30 22:29:53.390281 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:29:53.390254 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2d73912b_221e_4ff0_8144_af701cb79918.slice/crio-0c371786155fd3bcf9239e591da4c21fa463ff2fe249fda6aba162eabf54d8c7 WatchSource:0}: Error finding container 0c371786155fd3bcf9239e591da4c21fa463ff2fe249fda6aba162eabf54d8c7: Status 404 returned error can't find the container with id 0c371786155fd3bcf9239e591da4c21fa463ff2fe249fda6aba162eabf54d8c7 Oct 30 22:29:53.989922 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:29:53.989885 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-x6ll8" event={"ID":"2d73912b-221e-4ff0-8144-af701cb79918","Type":"ContainerStarted","Data":"698569c26e0cb2303dcfd52a2ca81ec908b2c6fbfb51bb0f1e43157a4ca3ccb6"} Oct 30 22:29:53.990132 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:29:53.989941 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-x6ll8" event={"ID":"2d73912b-221e-4ff0-8144-af701cb79918","Type":"ContainerStarted","Data":"0c371786155fd3bcf9239e591da4c21fa463ff2fe249fda6aba162eabf54d8c7"} Oct 30 22:29:53.990132 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:29:53.989975 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-x6ll8" Oct 30 22:29:54.103489 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:29:54.103439 4560 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-x6ll8" podStartSLOduration=2.103425409 podStartE2EDuration="2.103425409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:29:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:54.102052237 +0000 UTC m=+4569.656752015" watchObservedRunningTime="2026-10-30 22:29:54.103425409 +0000 UTC m=+4569.658125186" Oct 30 22:30:00.003284 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:30:00.003253 4560 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hgcw8/perf-node-gather-daemonset-x6ll8" Oct 30 22:30:00.176203 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:30:00.176169 4560 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-dxj5n"] Oct 30 22:30:00.179486 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:30:00.179468 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-dxj5n" Oct 30 22:30:00.229509 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:30:00.229475 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:30:00.229710 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:30:00.229534 4560 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:30:00.229710 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:30:00.229672 4560 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7b9qb\"" Oct 30 22:30:00.236026 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:30:00.235999 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-dxj5n"] Oct 30 22:30:00.265564 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:30:00.265484 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee843716-9e41-47db-97e6-65dc5e943f4c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889990-dxj5n\" (UID: \"ee843716-9e41-47db-97e6-65dc5e943f4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-dxj5n" Oct 30 22:30:00.265720 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:30:00.265629 4560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfhbl\" (UniqueName: \"kubernetes.io/projected/ee843716-9e41-47db-97e6-65dc5e943f4c-kube-api-access-tfhbl\") pod \"maas-api-key-cleanup-29889990-dxj5n\" (UID: \"ee843716-9e41-47db-97e6-65dc5e943f4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-dxj5n" Oct 30 22:30:00.366266 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:30:00.366226 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfhbl\" (UniqueName: \"kubernetes.io/projected/ee843716-9e41-47db-97e6-65dc5e943f4c-kube-api-access-tfhbl\") pod \"maas-api-key-cleanup-29889990-dxj5n\" (UID: \"ee843716-9e41-47db-97e6-65dc5e943f4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-dxj5n" Oct 30 22:30:00.366474 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:30:00.366290 4560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee843716-9e41-47db-97e6-65dc5e943f4c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889990-dxj5n\" (UID: \"ee843716-9e41-47db-97e6-65dc5e943f4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-dxj5n" Oct 30 22:30:00.366907 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:30:00.366885 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee843716-9e41-47db-97e6-65dc5e943f4c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889990-dxj5n\" (UID: \"ee843716-9e41-47db-97e6-65dc5e943f4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-dxj5n" Oct 30 22:30:00.424918 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:30:00.424888 4560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfhbl\" (UniqueName: \"kubernetes.io/projected/ee843716-9e41-47db-97e6-65dc5e943f4c-kube-api-access-tfhbl\") pod \"maas-api-key-cleanup-29889990-dxj5n\" (UID: \"ee843716-9e41-47db-97e6-65dc5e943f4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-dxj5n" Oct 30 22:30:00.493016 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:30:00.492983 4560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-dxj5n" Oct 30 22:30:00.515801 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:30:00.515729 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-756b9cccf4-cz4sx_017ad089-d65d-47a3-9b1e-10fbb9290e19/metrics-server/0.log" Oct 30 22:30:00.861088 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:30:00.860994 4560 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-dxj5n"] Oct 30 22:30:00.867877 ip-10-0-129-83 kubenswrapper[4560]: W1030 22:30:00.867850 4560 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee843716_9e41_47db_97e6_65dc5e943f4c.slice/crio-0f9ac124675f6579cdeac6791a8f248a7ba6eb4b3665dde6a97b3963c1b2dcc3 WatchSource:0}: Error finding container 0f9ac124675f6579cdeac6791a8f248a7ba6eb4b3665dde6a97b3963c1b2dcc3: Status 404 returned error can't find the container with id 0f9ac124675f6579cdeac6791a8f248a7ba6eb4b3665dde6a97b3963c1b2dcc3 Oct 30 22:30:01.019395 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:30:01.019357 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-dxj5n" event={"ID":"ee843716-9e41-47db-97e6-65dc5e943f4c","Type":"ContainerStarted","Data":"0f9ac124675f6579cdeac6791a8f248a7ba6eb4b3665dde6a97b3963c1b2dcc3"} Oct 30 22:30:01.355827 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:30:01.355799 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cwbdq_949d11d7-1fa3-4c6f-b08c-faa10f082e00/node-exporter/0.log" Oct 30 22:30:01.821592 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:30:01.821562 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cwbdq_949d11d7-1fa3-4c6f-b08c-faa10f082e00/kube-rbac-proxy/0.log" Oct 30 22:30:02.025923 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:30:02.025892 4560 generic.go:356] "Generic (PLEG): container finished" podID="ee843716-9e41-47db-97e6-65dc5e943f4c" containerID="fec50281f9404a89c618519163e117d67d46ec2b514ac6b13a49b0580625eb2b" exitCode=60 Oct 30 22:30:02.026367 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:30:02.025975 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-dxj5n" event={"ID":"ee843716-9e41-47db-97e6-65dc5e943f4c","Type":"ContainerDied","Data":"fec50281f9404a89c618519163e117d67d46ec2b514ac6b13a49b0580625eb2b"} Oct 30 22:30:02.026367 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:30:02.026177 4560 scope.go:122] "RemoveContainer" containerID="fec50281f9404a89c618519163e117d67d46ec2b514ac6b13a49b0580625eb2b" Oct 30 22:30:02.230334 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:30:02.230307 4560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cwbdq_949d11d7-1fa3-4c6f-b08c-faa10f082e00/init-textfile/0.log" Oct 30 22:30:03.031753 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:30:03.031711 4560 generic.go:356] "Generic (PLEG): container finished" podID="ee843716-9e41-47db-97e6-65dc5e943f4c" containerID="80c46986049319a83333069a31300a83cc4ff7a2db6ca1e2a8795b86404c76ae" exitCode=60 Oct 30 22:30:03.032175 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:30:03.031770 4560 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-dxj5n" event={"ID":"ee843716-9e41-47db-97e6-65dc5e943f4c","Type":"ContainerDied","Data":"80c46986049319a83333069a31300a83cc4ff7a2db6ca1e2a8795b86404c76ae"} Oct 30 22:30:03.032175 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:30:03.031824 4560 scope.go:122] "RemoveContainer" containerID="fec50281f9404a89c618519163e117d67d46ec2b514ac6b13a49b0580625eb2b" Oct 30 22:30:03.032175 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:30:03.032027 4560 scope.go:122] "RemoveContainer" containerID="80c46986049319a83333069a31300a83cc4ff7a2db6ca1e2a8795b86404c76ae" Oct 30 22:30:03.032319 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:30:03.032297 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889990-dxj5n_odh-ai-gateway-infra(ee843716-9e41-47db-97e6-65dc5e943f4c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-dxj5n" podUID="ee843716-9e41-47db-97e6-65dc5e943f4c" Oct 30 22:30:04.037286 ip-10-0-129-83 kubenswrapper[4560]: I1030 22:30:04.037260 4560 scope.go:122] "RemoveContainer" containerID="80c46986049319a83333069a31300a83cc4ff7a2db6ca1e2a8795b86404c76ae" Oct 30 22:30:04.037688 ip-10-0-129-83 kubenswrapper[4560]: E1030 22:30:04.037485 4560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889990-dxj5n_odh-ai-gateway-infra(ee843716-9e41-47db-97e6-65dc5e943f4c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-dxj5n" podUID="ee843716-9e41-47db-97e6-65dc5e943f4c"